00000800:00000100:2.1F:1787695771.749592:0:2390:0:(socklnd_cb.c:2294:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.201.136@tcp (192.168.201.136:988) the network or that node may be down. 00000100:00100000:3.0:1787695890.342516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695890/real 1787695890] req@ffff9c5cc35af480 x1874531126020608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695890.342545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126020608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695890.342555:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126020608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695915 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695890.342563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695890.342566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695890.342571:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126020608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695915 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695890.342578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126020608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695890.342584:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126020608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695915 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695890.342593:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695890.853513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:3.0:1787695890.981547:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787695891.877508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:3.0:1787695892.005564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787695892.261531:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298465647, at 4298465664 00000100:00100000:3.0:1787695892.901523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:3.0:1787695893.029535:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787695893.925513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:3.0:1787695894.053566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787695894.949520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:3.0:1787695895.077576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787695895.461586:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695895.461610:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787695895.461613:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695895.461639:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 3796 00000100:00080000:3.0:1787695895.461648:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695895.461651:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461654:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461656:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787695895.461660:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787695895.461663:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461665:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461667:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787695895.461669:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461679:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787695895.461699:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126020736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695895.461710:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126020736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695895.461732:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695895.461735:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787695895.461736:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695895.461740:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3796 00000100:00080000:3.0:1787695895.461744:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695895.461746:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461749:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461752:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787695895.461754:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787695895.461765:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461769:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461772:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787695895.461775:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695895.461786:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695895.461794:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695895.461810:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126020864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695895.461826:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126020864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695895.461833:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695895.461835:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787695895.461837:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695895.461840:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3796 00000100:00080000:3.0:1787695895.461844:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695895.461846:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461851:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461854:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787695895.461858:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787695895.461867:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461875:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461882:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787695895.461889:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695895.461895:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695895.461901:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695895.461911:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126020992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695895.461928:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126020992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695895.461936:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695895.461939:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787695895.461941:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695895.461944:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3796 00000100:00080000:3.0:1787695895.461949:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695895.461954:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461960:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461967:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787695895.461972:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787695895.461977:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.461981:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.461983:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787695895.461987:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695895.461994:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695895.462000:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695895.462016:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126021120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695895.462029:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126021120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695895.462062:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695895.462065:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787695895.462067:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695895.462088:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3796 00000100:00080000:3.0:1787695895.462094:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695895.462100:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.462107:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.462111:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787695895.462115:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787695895.462120:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695895.462125:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695895.462131:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787695895.462135:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695895.462142:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695895.462149:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695895.462162:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126021248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695895.462170:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126021248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695895.462182:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (3806) 00000100:00100000:3.0:1787695895.973516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:3.0:1787695896.101600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787695896.997540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:3.0:1787695897.125555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787695897.381534:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298470767, at 4298470784 00000100:00100000:3.0:1787695898.021507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695900.581741:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.581759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126021376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695900.581809:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.581842:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695900.581888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695900.581910:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.581950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695900/real 1787695900] req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.581992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582013:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695900.582026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695900.582034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695900.582044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126021376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695900.582065:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126021376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695900.582079:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695900.582101:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695900.582165:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582179:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695900.582196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695900.582205:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582217:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695900.582225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695900/real 1787695900] req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695900.582246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695900.582248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695900.582251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695900.582259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695900.582276:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582289:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695900.582298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695900.582303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582311:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695900.582314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695900/real 1787695900] req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695900.582334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695900.582336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695900.582338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695900.582347:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695900.582362:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582366:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695900.582378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695900.582383:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582390:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695900.582394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695900/real 1787695900] req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582407:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695900.582412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695900.582414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695900.582436:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582442:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695900.582446:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126021504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695900.582452:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695900.582457:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126021632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695900.582462:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695900.582465:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126021760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695900.582469:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695900.582481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695900.582497:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582503:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695900.582512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695900.582517:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582523:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695900.582527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695900/real 1787695900] req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695900.582559:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695900.582564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695900.582566:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695900.582568:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695900.582573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126021888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695900.582578:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126021888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695900.582591:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695902.117717:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695902.501546:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298475887, at 4298475904 00000100:00100000:3.0:1787695903.141544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695904.165516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695905.189551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695905.701763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.701782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126022016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695905.701829:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.701852:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695905.701896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695905.701919:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.701970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695905/real 1787695905] req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.701991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702028:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695905.702054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695905.702066:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695905.702080:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126022016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695905.702124:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126022016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695905.702143:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695905.702166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695905.702195:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702203:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695905.702218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695905.702226:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702236:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695905.702243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695905/real 1787695905] req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702266:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695905.702274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695905.702277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695905.702289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695905.702308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695905.702325:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702330:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695905.702340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695905.702345:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702352:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695905.702358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695905/real 1787695905] req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702389:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695905.702398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695905.702402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695905.702406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695905.702473:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695905.702499:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702508:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695905.702520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695905.702527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702538:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695905.702544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695905/real 1787695905] req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702565:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695905.702570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695905.702573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695905.702575:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695905.702590:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702596:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695905.702605:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702610:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695905.702618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695905.702623:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695930 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702629:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695905.702632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695905/real 1787695905] req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702640:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695905.702645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695905.702650:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695905.702652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695905.702654:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695905.702658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695905.702662:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126022144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695905.702675:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695905.702680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126022272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695905.702684:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695905.702688:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126022400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695905.702692:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695905.702695:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126022528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695930 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695905.702699:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695906.213515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695907.237547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695907.621489:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298481007, at 4298481024 00000100:00100000:3.0:1787695908.261527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695909.285530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695910.309515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695910.821727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.821748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126022656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695910.821780:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.821796:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695910.821846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695910.821882:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.821946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695910/real 1787695910] req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.821976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.821992:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695910.822016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695910.822024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695910.822036:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822057:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126022656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695910.822063:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126022656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695910.822072:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695910.822084:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695910.822103:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822108:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695910.822119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695910.822124:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822152:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695910.822157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695910/real 1787695910] req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695910.822189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695910.822193:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695910.822197:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695910.822212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695910.822241:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822250:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695910.822263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695910.822271:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822282:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695910.822292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695910/real 1787695910] req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695910.822312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695910.822314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695910.822316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126022912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695910.822329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695910.822364:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822373:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695910.822388:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695910.822396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822407:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695910.822413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695910/real 1787695910] req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822459:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695910.822470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695910.822472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695910.822474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822479:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695910.822483:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695910.822498:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822503:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695910.822514:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695910.822521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695935 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822531:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695910.822537:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695910/real 1787695910] req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822550:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695910.822567:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695910.822575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695910.822579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695910.822583:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695910.822591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695910.822598:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126022784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695910.822607:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695910.822612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126022912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695910.822617:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695910.822620:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126023040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695910.822625:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695910.822628:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126023168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695935 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695910.822632:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695911.333548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695912.357527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695912.741539:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298486127, at 4298486144 00000100:00100000:3.0:1787695913.381527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695914.405565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695915.429543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695915.941722:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.941744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126023296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695915.941773:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.941808:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695915.941858:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695915.941882:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.941919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695915/real 1787695915] req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.941951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.941964:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695915.941974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695915.941980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695915.941988:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126023296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695915.942014:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126023296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695915.942037:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695915.942056:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695915.942090:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942100:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695915.942125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695915.942139:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942153:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695915.942162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695915/real 1787695915] req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695915.942185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695915.942187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695915.942189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695915.942205:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942211:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695915.942221:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942226:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695915.942234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695915.942239:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942245:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695915.942249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695915/real 1787695915] req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695915.942285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695915.942289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695915.942294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695915.942309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695915.942338:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942346:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695915.942360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695915.942368:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942378:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695915.942383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695915/real 1787695915] req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695915.942402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695915.942404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695915.942406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695915.942448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126023424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695915.942459:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695915.942472:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126023552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695915.942481:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695915.942486:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126023680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695915.942491:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695915.942497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695915.942512:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942517:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695915.942526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695915.942531:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695940 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942541:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695915.942547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695915/real 1787695915] req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695915.942570:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695915.942578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695915.942582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695915.942590:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695915.942596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126023808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695915.942600:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126023808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695940 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695915.942605:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695916.453560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695917.477575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695917.861523:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298491247, at 4298491264 00000100:00100000:3.0:1787695918.501536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695919.525525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695920.549558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787695921.061540:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695921.061549:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787695921.061553:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695921.061561:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 3822 00000100:00080000:3.0:1787695921.061571:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695921.061574:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061577:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061580:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787695921.061584:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787695921.061587:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061589:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061591:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787695921.061593:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061604:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787695921.061640:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126023936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695921.061660:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126023936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695921.061679:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695921.061682:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787695921.061684:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695921.061687:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3822 00000100:00080000:3.0:1787695921.061690:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695921.061692:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061694:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061695:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787695921.061698:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787695921.061701:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061703:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061705:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787695921.061707:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695921.061715:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695921.061721:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695921.061733:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126024064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695921.061752:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126024064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695921.061759:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695921.061761:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787695921.061769:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695921.061774:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3822 00000100:00080000:3.0:1787695921.061781:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695921.061791:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061799:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061802:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787695921.061806:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787695921.061810:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061813:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061816:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787695921.061819:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695921.061827:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695921.061834:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695921.061846:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126024192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695921.061863:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126024192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695921.061870:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695921.061872:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787695921.061873:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695921.061877:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3822 00000100:00080000:3.0:1787695921.061881:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695921.061885:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061891:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061895:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787695921.061899:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787695921.061909:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.061912:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.061913:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787695921.061918:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695921.061925:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695921.061931:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695921.061942:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126024320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695921.061965:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126024320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695921.061977:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695921.061984:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787695921.061988:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695921.061998:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3822 00000100:00080000:3.0:1787695921.062004:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695921.062013:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.062019:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.062027:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787695921.062040:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787695921.062051:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695921.062062:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695921.062071:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787695921.062077:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695921.062084:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695921.062094:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695921.062111:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126024448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695921.062156:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126024448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695921.062206:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (3832) 00000100:00100000:3.0:1787695921.573520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695922.597539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695922.981510:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298496367, at 4298496384 00000100:00100000:3.0:1787695923.621519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695924.645471:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695925.669544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787695926.181531:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695926.181557:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787695926.181562:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695926.181573:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 3827 00000100:00080000:3.0:1787695926.181583:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695926.181588:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.181592:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.181596:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787695926.181601:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787695926.181605:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.181608:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.181611:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787695926.181615:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787695926.181630:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787695926.181659:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126024576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695926.181676:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126024576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695926.181715:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695926.181720:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787695926.181721:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695926.181728:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3827 00000100:00080000:3.0:1787695926.181733:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695926.181737:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.181741:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.181745:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787695926.181749:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787695926.181759:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.181764:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.181769:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787695926.181777:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695926.181788:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695926.181795:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695926.181827:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126024704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695926.181862:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126024704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695926.181874:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695926.181879:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787695926.181882:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695926.181887:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3827 00000100:00080000:3.0:1787695926.181892:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695926.181905:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.181910:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.181914:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787695926.181918:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787695926.181925:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.181935:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.181941:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787695926.181948:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695926.181960:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695926.181969:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695926.181989:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126024832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695926.182009:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126024832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695926.182032:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695926.182037:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787695926.182040:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695926.182048:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3827 00000100:00080000:3.0:1787695926.182059:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695926.182067:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.182072:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.182078:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787695926.182083:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787695926.182124:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.182130:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.182133:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787695926.182146:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695926.182154:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695926.182160:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695926.182172:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126024960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695926.182184:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126024960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695926.182205:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695926.182209:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787695926.182211:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695926.182214:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3827 00000100:00080000:3.0:1787695926.182219:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695926.182223:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.182227:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.182230:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787695926.182233:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787695926.182237:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695926.182241:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695926.182244:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787695926.182250:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695926.182259:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695926.182267:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695926.182283:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126025088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695926.182300:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126025088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695926.182327:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (3837) 00000100:00100000:3.0:1787695926.693514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695927.717581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695927.717611:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695928.101556:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298501487, at 4298501504 00000100:00100000:3.0:1787695928.741545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695929.765537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695930.789535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787695931.301507:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695931.301515:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787695931.301520:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695931.301531:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 3832 00000100:00080000:3.0:1787695931.301542:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695931.301546:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301551:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301556:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787695931.301561:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787695931.301565:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301569:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301572:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787695931.301575:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301586:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787695931.301609:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126025216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695931.301621:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126025216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695931.301641:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695931.301656:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787695931.301658:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695931.301661:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3832 00000100:00080000:3.0:1787695931.301665:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695931.301668:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301669:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301671:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787695931.301674:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787695931.301676:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301678:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301680:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787695931.301682:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695931.301697:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695931.301707:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695931.301724:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126025344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695931.301736:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126025344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695931.301747:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695931.301756:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787695931.301759:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695931.301765:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3832 00000100:00080000:3.0:1787695931.301773:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695931.301780:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301789:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301799:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787695931.301802:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787695931.301818:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301821:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301824:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787695931.301831:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695931.301841:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695931.301847:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695931.301864:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126025472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695931.301883:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126025472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695931.301913:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695931.301917:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787695931.301919:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695931.301927:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3832 00000100:00080000:3.0:1787695931.301933:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695931.301940:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301967:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.301972:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787695931.301978:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787695931.301987:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.301994:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.302008:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787695931.302015:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695931.302026:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695931.302041:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695931.302061:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126025600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695931.302073:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126025600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695931.302086:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695931.302096:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787695931.302101:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695931.302110:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3832 00000100:00080000:3.0:1787695931.302118:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695931.302121:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.302124:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.302127:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787695931.302133:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787695931.302143:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695931.302151:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695931.302158:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787695931.302165:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695931.302172:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695931.302180:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695931.302198:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126025728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695931.302216:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126025728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695931.302254:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (3842) 00000100:00100000:3.0:1787695931.813543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695932.837526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695933.221473:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298506607, at 4298506624 00000100:00000040:3.0:1787695938.341531:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298511727, at 4298511744 00000100:00100000:3.0:1787695940.005671:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695942.053552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695943.077531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695943.461566:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298516847, at 4298516864 00000100:00100000:3.0:1787695944.101537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695945.125496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695946.149538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695946.661899:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.661930:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126027136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695946.661974:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662001:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695946.662053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695946.662080:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695946/real 1787695946] req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695946.662215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695946.662226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695946.662237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126027136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695946.662275:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126027136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695946.662287:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695946.662306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695946.662340:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662350:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695946.662361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695946.662368:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662380:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695946.662386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695946/real 1787695946] req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662441:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695946.662451:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695946.662455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695946.662460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695946.662475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695946.662497:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662506:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695946.662517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695946.662524:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662534:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695946.662539:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695946/real 1787695946] req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662561:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695946.662576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695946.662580:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695946.662584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695946.662599:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695946.662623:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662632:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695946.662643:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695946.662650:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662660:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695946.662666:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695946/real 1787695946] req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662679:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695946.662698:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695946.662706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695946.662710:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662720:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695946.662726:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662753:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695946.662767:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662774:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695946.662786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695946.662793:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662803:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695946.662872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695946/real 1787695946] req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695946.662897:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695946.662905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695946.662908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695946.662913:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695946.662927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695946.662934:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126027264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695946.662942:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695946.662949:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126027392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695946.662956:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695946.662961:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126027520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695946.662969:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695946.662974:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126027648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695946.662981:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695947.173595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695948.197590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695948.581494:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298521967, at 4298521984 00000100:00100000:3.0:1787695949.221547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695950.245525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695951.269526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695951.781766:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.781786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126027776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695951.781825:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.781863:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695951.781911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695951.781935:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.781975:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695951/real 1787695951] req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782027:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695951.782040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695951.782047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695951.782055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126027776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695951.782084:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126027776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695951.782094:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695951.782108:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695951.782126:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782141:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695951.782151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695951.782155:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782164:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695951.782168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695951/real 1787695951] req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782183:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695951.782188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695951.782190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695951.782192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126027904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695951.782200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695951.782213:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782218:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695951.782230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695951.782234:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782241:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695951.782245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695951/real 1787695951] req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695951.782264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695951.782265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695951.782268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695951.782275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695951.782289:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782293:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695951.782299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695951.782303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782318:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695951.782322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695951/real 1787695951] req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695951.782340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695951.782342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695951.782345:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695951.782353:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782358:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695951.782365:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782370:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695951.782376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695951.782379:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695976 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782385:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695951.782395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695951/real 1787695951] req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695951.782409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695951.782414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695951.782415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695951.782438:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695951.782444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695951.782448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126027904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695951.782453:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695951.782458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126028032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695951.782463:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695951.782467:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126028160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695951.782472:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695951.782475:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126028288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695976 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695951.782479:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695952.293561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695953.317539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695953.701599:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298527087, at 4298527104 00000100:00100000:3.0:1787695954.341569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695955.365529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695956.389526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787695956.901780:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.901815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126028416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695956.901844:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.901873:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695956.901907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695956.901928:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.901987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695956/real 1787695956] req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902053:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695956.902069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695956.902075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695956.902081:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126028416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695956.902134:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126028416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695956.902157:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695956.902181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695956.902228:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902253:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695956.902269:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695956.902278:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902287:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695956.902292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695956/real 1787695956] req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695956.902321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695956.902324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695956.902332:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695956.902342:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695956.902356:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902362:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695956.902376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695956.902381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902388:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695956.902392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695956/real 1787695956] req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695956.902416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695956.902435:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695956.902439:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695956.902459:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695956.902476:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902482:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695956.902492:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695956.902500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902508:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695956.902512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695956/real 1787695956] req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695956.902530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695956.902532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695956.902535:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695956.902547:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126028544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695956.902553:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695956.902574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126028672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695956.902580:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695956.902584:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126028800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695956.902588:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695956.902594:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695956.902610:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902615:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695956.902623:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695956.902628:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787695981 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902634:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695956.902638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695956/real 1787695956] req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695956.902651:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695956.902662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695956.902664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695956.902667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695956.902672:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126028928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695956.902675:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126028928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787695981 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695956.902680:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695958.437562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695958.821548:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298532207, at 4298532224 00000100:00100000:3.0:1787695959.461549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695960.485511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695961.509563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787695962.021573:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695962.021584:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787695962.021590:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695962.021599:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 3863 00000100:00080000:3.0:1787695962.021608:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695962.021611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.021614:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.021618:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787695962.021623:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787695962.021627:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.021630:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.021633:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787695962.021655:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787695962.021667:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787695962.021688:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126029056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695962.021700:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126029056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695962.021720:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695962.021723:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787695962.021725:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695962.021728:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3863 00000100:00080000:3.0:1787695962.021732:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695962.021735:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.021737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.021739:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787695962.021742:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787695962.021745:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.021747:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.021749:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787695962.021755:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695962.021766:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695962.021772:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695962.021783:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126029184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695962.021792:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126029184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695962.021832:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695962.021839:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787695962.021841:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695962.021845:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3863 00000100:00080000:3.0:1787695962.021862:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695962.021865:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.021871:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.021877:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787695962.021880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787695962.021884:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.021891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.021898:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787695962.021905:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695962.021911:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787695962.021918:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695962.021932:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126029312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695962.021962:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126029312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695962.021978:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695962.021981:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787695962.021983:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695962.021989:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3863 00000100:00080000:3.0:1787695962.021996:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695962.022013:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.022023:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.022028:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787695962.022034:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787695962.022044:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.022054:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.022062:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787695962.022070:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695962.022083:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695962.022093:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695962.022106:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126029440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695962.022137:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126029440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695962.022151:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787695962.022154:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787695962.022157:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787695962.022164:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3863 00000100:00080000:3.0:1787695962.022170:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787695962.022172:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.022178:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.022182:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787695962.022185:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787695962.022191:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787695962.022194:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787695962.022195:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787695962.022206:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787695962.022213:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787695962.022217:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695962.022231:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126029568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787695962.022245:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126029568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695962.022280:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (3873) 00000100:00100000:3.0:1787695962.533506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695963.557562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695963.941546:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298537327, at 4298537344 00000100:00100000:3.0:1787695964.581580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695965.605549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695966.629527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695967.653520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695968.677535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695969.061525:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298542447, at 4298542464 00000100:00100000:3.0:1787695969.701547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695970.725541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695970.725682:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695971.749527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695971.749572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695972.773555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695973.797559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695974.181538:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298547567, at 4298547584 00000100:00100000:3.0:1787695974.821559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695975.845509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695976.869535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695977.893489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695978.917583:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695979.301540:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298552687, at 4298552704 00000100:00100000:3.0:1787695980.965553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695981.989514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695983.013572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695984.037498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787695984.037562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695984.421530:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298557807, at 4298557824 00000100:00100000:3.0:1787695985.061518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695986.085561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695987.109547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695988.133540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695989.157532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695989.541495:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298562927, at 4298562944 00000100:00100000:3.0:1787695990.181551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695991.205518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695992.229566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695992.741724:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126032896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695992.741739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126032896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695992.741774:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126032896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696017 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695992.741790:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695992.741874:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126033024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695992.741891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695992.741910:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126033024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696017 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695992.741964:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695992.741985:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787695992.742002:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126033152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695992.742020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695992.742046:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126033152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696017 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695992.742060:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695992.742077:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787695992.742094:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126033280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695992.742113:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695992.742136:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126033280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696017 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695992.742159:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695992.742175:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787695992.742982:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126033408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695992.742994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695992.743011:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126033408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696017 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695992.743019:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695992.743035:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695993.253520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695994.277526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695994.661525:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298568047, at 4298568064 00000100:00100000:3.0:1787695995.301481:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695995.813765:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695992/real 1787695995] req@ffff9c5cc35ae300 x1874531126033024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.813800:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126033024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695995.813835:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126033024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695995.813849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695995.813857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695995.813863:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126033024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.813872:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033024:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787695995.813878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695992/real 1787695995] req@ffff9c5cc35aea00 x1874531126032896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.813900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126032896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695995.813909:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126032896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695995.813916:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695995.813919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695995.813923:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126032896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.813956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126032896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695995.813963:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126033024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695995.813976:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695995.813986:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126032896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695995.813992:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787695995.813999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695992/real 1787695995] req@ffff9c5cc35ace00 x1874531126033408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.814011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126033408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695995.814018:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126033408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695995.814026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695995.814035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695995.814039:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126033408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.814046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033408:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787695995.814053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695992/real 1787695995] req@ffff9c5cc35ae680 x1874531126033280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.814064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126033280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695995.814078:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126033280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695995.814086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695995.814089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695995.814093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126033280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.814099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033280:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787695995.814105:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695992/real 1787695995] req@ffff9c5cc35ad880 x1874531126033152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.814116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126033152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695995.814124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126033152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695995.814131:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695995.814135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695995.814139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126033152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695995.814146:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695995.814152:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126033408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695995.814159:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695995.814170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126033280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695995.814177:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695995.814182:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126033152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696017 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695995.814189:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787695996.325525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695997.349509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695997.861761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.861779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126033536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695997.861816:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.861833:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695997.861894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695997.861921:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.861965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695997/real 1787695997] req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.861993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.862014:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695997.862035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695997.862044:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695997.862061:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.862076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126033536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787695997.862084:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126033536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695997.862101:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787695997.862128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.862167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695997.862188:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.862206:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695997.862226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695997.862240:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.862254:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787695997.862261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695997/real 1787695997] req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.862274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.862283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695997.862290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695997.862303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695997.862307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.862314:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695997.862916:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.862927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695997.862948:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.862956:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695997.862974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695997.862983:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.862993:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787695997.863000:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695997/real 1787695997] req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.863018:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695997.863026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695997.863029:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695997.863032:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863050:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787695997.863056:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695997.863078:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.863084:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695997.863096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695997.863102:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.863112:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787695997.863118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695997/real 1787695997] req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.863137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695997.863144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695997.863148:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695997.863151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126033920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695997.863172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126033664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695997.863182:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695997.863189:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126033792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695997.863196:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787695997.863201:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126033920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695997.863214:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787695997.863227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695997.863250:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.863257:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787695997.863268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787695997.863274:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696022 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.863283:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787695997.863288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787695997/real 1787695997] req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787695997.863305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787695997.863320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787695997.863323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787695997.863327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787695997.863333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787695997.863338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126034048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696022 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787695997.863345:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787695998.373538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787695999.397514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787695999.781916:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298573167, at 4298573184 00000100:00100000:3.0:1787696000.421527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696001.445485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696002.469535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696002.981868:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.981897:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126034176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696002.981937:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.981951:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696002.981998:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696002.982036:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696002/real 1787696002] req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982127:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696002.982155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696002.982162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696002.982170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126034176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696002.982201:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126034176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696002.982215:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696002.982240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696002.982279:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982298:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696002.982312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696002.982323:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982338:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696002.982349:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696002/real 1787696002] req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696002.982407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696002.982414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696002.982450:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696002.982465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696002.982487:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982493:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696002.982505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696002.982512:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982522:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696002.982528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696002/real 1787696002] req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982560:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696002.982568:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696002.982571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696002.982575:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696002.982588:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696002.982610:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982618:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696002.982628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696002.982635:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982644:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696002.982648:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696002/real 1787696002] req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982682:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696002.982689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696002.982692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696002.982695:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696002.982709:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126034304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696002.982717:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696002.982723:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126034432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696002.982731:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696002.982736:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126034560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696002.982742:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696002.982751:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696002.982801:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982808:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696002.982820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696002.982828:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696027 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982837:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696002.982842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696002/real 1787696002] req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982853:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696002.982861:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696002.982869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696002.982872:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696002.982875:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696002.982882:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696002.982888:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126034688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696027 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696002.982897:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696003.493517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696004.517518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696004.901517:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298578287, at 4298578304 00000100:00100000:3.0:1787696005.541558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696006.565505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696007.589542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696008.101711:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.101739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126034816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696008.101774:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.101793:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696008.101827:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696008.101845:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.101882:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696008/real 1787696008] req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.101912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.101926:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696008.101948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696008.101956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696008.101964:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.101981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126034816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696008.102006:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126034816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696008.102029:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696008.102056:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696008.102119:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102128:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696008.102143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696008.102150:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102161:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696008.102166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696008/real 1787696008] req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696008.102188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696008.102190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696008.102193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126034944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696008.102211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102218:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696008.102226:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102231:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696008.102240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696008.102244:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102251:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696008.102254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696008/real 1787696008] req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696008.102272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696008.102274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696008.102276:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696008.102290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696008.102306:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102311:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696008.102317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696008.102321:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102327:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696008.102331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696008/real 1787696008] req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696008.102349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696008.102350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696008.102353:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696008.102360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126034944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696008.102370:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696008.102375:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126035072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696008.102379:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696008.102383:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126035200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696008.102387:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696008.102393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696008.102405:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102410:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696008.102436:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696008.102442:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102448:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696008.102452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696008/real 1787696008] req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696008.102466:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696008.102479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696008.102482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696008.102484:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696008.102489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696008.102493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126035328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696008.102498:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696008.613564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696009.637542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696010.021547:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298583407, at 4298583424 00000100:00100000:3.0:1787696010.661548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696010.789580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696011.685522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696011.813570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696012.837626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787696013.221869:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.221888:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126035456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696013.221928:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.221945:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696013.221990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696013.222011:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696013/real 1787696013] req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696013.222113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696013.222120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696013.222124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126035456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696013.222175:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126035456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696013.222200:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696013.222236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696013.222288:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222295:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696013.222307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696013.222313:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222338:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696013.222346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696013/real 1787696013] req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222371:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696013.222379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696013.222381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696013.222392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696013.222402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696013.222443:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222452:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696013.222476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696013.222485:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222497:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696013.222510:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696013/real 1787696013] req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696013.222539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696013.222542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696013.222544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696013.222554:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696013.222570:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222579:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696013.222601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696013.222607:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222615:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696013.222619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696013/real 1787696013] req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222643:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696013.222649:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696013.222651:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696013.222653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222658:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696013.222662:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126035584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696013.222668:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696013.222673:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126035712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696013.222678:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696013.222681:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126035840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696013.222686:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696013.222691:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696013.222711:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222716:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696013.222724:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696013.222729:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222735:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696013.222739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696013/real 1787696013] req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222747:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696013.222753:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696013.222758:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696013.222760:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696013.222762:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696013.222766:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126035968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696013.222770:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126035968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696013.222774:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00010000:00010000:3.0:1787696013.861540:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1787696014.885552:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787696015.141523:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298588527, at 4298588544 00000100:00100000:3.0:1787696015.781583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696015.909566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696016.805490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696016.933505:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696017.829552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696017.957522:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696018.853512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696018.981581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696019.877562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696020.261499:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298593647, at 4298593664 00000100:00100000:3.0:1787696020.901496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696021.925588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696022.949513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696023.461719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.461761:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126036736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696023.461811:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.461835:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696023.461884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696023.461906:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.461940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696023/real 1787696023] req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.461967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462016:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696023.462042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696023.462050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696023.462057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126036736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696023.462076:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126036736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696023.462087:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696023.462103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462114:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126036864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696023.462130:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462139:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696023.462152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696023.462160:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462171:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696023.462178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696023/real 1787696023] req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696023.462226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696023.462229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696023.462234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126036864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696023.462249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126036992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696023.462272:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462280:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696023.462294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696023.462303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462313:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696023.462319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696023/real 1787696023] req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696023.462368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696023.462372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696023.462375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126036992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696023.462391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696023.462457:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462467:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696023.462480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696023.462488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462499:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696023.462505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696023/real 1787696023] req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462542:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696023.462549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696023.462553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696023.462556:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696023.462571:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696023.462593:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462601:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696023.462614:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696023.462622:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462631:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696023.462637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696023/real 1787696023] req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696023.462665:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696023.462674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696023.462677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696023.462681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696023.462689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696023.462695:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126036864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696023.462703:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696023.462710:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126036992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696023.462718:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696023.462723:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126037120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696023.462730:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696023.462736:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126037248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696023.462744:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696023.973483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696024.997533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696025.381527:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298598767, at 4298598784 00000100:00100000:3.0:1787696026.021541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696027.045543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696028.069508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696028.581755:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.581785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126037376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696028.581822:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.581845:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696028.581881:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696028.581901:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.581931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696028/real 1787696028] req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.581946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.581959:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696028.581985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696028.581991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696028.581999:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126037376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696028.582027:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126037376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696028.582063:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696028.582086:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696028.582132:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582154:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696028.582177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696028.582193:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582220:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696028.582233:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696028/real 1787696028] req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582264:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696028.582270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696028.582275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696028.582285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696028.582344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696028.582373:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582382:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696028.582397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696028.582406:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582416:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696028.582447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696028/real 1787696028] req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582472:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696028.582481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696028.582485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696028.582489:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582497:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696028.582504:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126037504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696028.582521:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696028.582530:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126037632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696028.582538:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696028.582548:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582558:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696028.582574:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582583:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696028.582597:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696028.582605:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582615:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696028.582622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696028/real 1787696028] req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696028.582653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696028.582657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696028.582661:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696028.582686:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696028.582710:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582719:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696028.582734:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696028.582742:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582752:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696028.582759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696028/real 1787696028] req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582772:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696028.582782:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696028.582789:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696028.582792:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696028.582797:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696028.582805:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126037888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696028.582818:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126037760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696028.582826:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696028.582832:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126037888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696028.582840:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696029.093518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696030.117515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696030.501522:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298603887, at 4298603904 00000100:00100000:3.0:1787696031.141562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696032.165505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696033.189528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696033.701702:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.701726:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126038016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696033.701759:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.701777:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696033.701817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696033.701842:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.701892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696033/real 1787696033] req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.701935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.701986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696033.702017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696033.702026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696033.702033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126038016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696033.702051:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126038016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696033.702062:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696033.702078:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696033.702105:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702113:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696033.702128:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696033.702137:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702148:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696033.702155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696033/real 1787696033] req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696033.702201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696033.702205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696033.702209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696033.702224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696033.702260:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702268:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696033.702281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696033.702290:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702299:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696033.702305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696033/real 1787696033] req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696033.702349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696033.702353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696033.702357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696033.702372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696033.702394:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702402:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696033.702414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696033.702452:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702462:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696033.702469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696033/real 1787696033] req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696033.702511:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696033.702515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696033.702519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696033.702533:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696033.702555:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702563:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696033.702576:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696033.702583:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696058 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702593:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696033.702599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696033/real 1787696033] req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696033.702629:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696033.702638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696033.702642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696033.702646:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696033.702655:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696033.702662:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126038144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696033.702671:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696033.702679:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126038272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696033.702687:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696033.702699:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126038400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696033.702707:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696033.702712:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126038528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696058 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696033.702720:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696034.213486:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696035.237529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696035.621502:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298609007, at 4298609024 00000100:00100000:3.0:1787696036.261481:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696037.285536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696038.821697:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.821727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126038656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696038.821762:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.821786:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696038.821825:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696038.821857:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.821900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696038/real 1787696038] req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.821934:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.821960:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696038.821984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696038.821991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696038.821999:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126038656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696038.822027:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126038656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696038.822053:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696038.822092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696038.822140:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822178:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696038.822198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696038.822210:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696038.822230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696038/real 1787696038] req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696038.822261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696038.822264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696038.822269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822278:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696038.822283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696038.822320:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822328:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696038.822339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696038.822345:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822356:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696038.822361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696038/real 1787696038] req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696038.822392:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696038.822396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696038.822400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126038912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696038.822415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696038.822466:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822474:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696038.822485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696038.822492:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822502:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696038.822508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696038/real 1787696038] req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696038.822538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696038.822540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696038.822544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696038.822558:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126038784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696038.822567:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696038.822573:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126038912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696038.822593:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696038.822600:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126039040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696038.822607:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696038.822615:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822625:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696038.822638:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822646:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696038.822657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696038.822665:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696063 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822674:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696038.822679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696038/real 1787696038] req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822691:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696038.822701:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696038.822709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696038.822712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696038.822716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696038.822730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696038.822736:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126039168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696063 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696038.822744:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696040.741558:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298614127, at 4298614144 00000100:00000040:3.0:1787696043.941787:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.941812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126039296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696043.941854:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.941867:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696043.941916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696043.941982:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696043/real 1787696043] req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942132:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696043.942160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696043.942166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696043.942181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126039296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696043.942194:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126039296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696043.942203:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696043.942218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696043.942238:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942243:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696043.942254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696043.942262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942273:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696043.942280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696043/real 1787696043] req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942303:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696043.942311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696043.942319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696043.942323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696043.942344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942350:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696043.942362:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942367:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696043.942376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696043.942382:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942397:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696043.942401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696043/real 1787696043] req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942409:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696043.942448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696043.942450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696043.942453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696043.942469:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696043.942487:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942492:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696043.942501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696043.942506:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942513:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696043.942517:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696043/real 1787696043] req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942535:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696043.942544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696043.942548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696043.942552:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696043.942569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696043.942603:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942608:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696043.942617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696043.942625:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696068 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942634:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696043.942641:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696043/real 1787696043] req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942654:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696043.942665:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696043.942673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696043.942675:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696043.942678:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696043.942684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126039808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696043.942687:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126039424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696043.942706:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696043.942713:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126039552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696043.942721:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696043.942727:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126039680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696043.942735:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696043.942740:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126039808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696068 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696043.942748:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696044.453696:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696045.477571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696045.861513:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298619247, at 4298619264 00000100:00100000:3.0:1787696046.501549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696047.525565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696047.525736:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696048.549536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696049.061639:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.061654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126039936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696049.061681:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.061692:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696049.061731:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696049.061750:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.061791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696049/real 1787696049] req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.061815:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.061843:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696049.061862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696049.061869:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696049.061879:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.061897:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126039936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696049.061908:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126039936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696049.061920:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696049.061937:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.061954:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696049.061986:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.061995:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696049.062008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696049.062015:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062031:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696049.062035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696049/real 1787696049] req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062047:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696049.062052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696049.062054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696049.062056:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696049.062063:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696049.062094:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062101:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696049.062112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696049.062119:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062126:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696049.062135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696049/real 1787696049] req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696049.062158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696049.062160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696049.062163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696049.062170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696049.062183:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062187:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696049.062194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696049.062199:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062204:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696049.062207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696049/real 1787696049] req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696049.062237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696049.062239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696049.062241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696049.062248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126040064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696049.062253:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696049.062256:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126040192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696049.062261:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696049.062264:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126040320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696049.062267:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696049.062272:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696049.062288:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062293:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696049.062299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696049.062303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062308:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696049.062311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696049/real 1787696049] req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696049.062323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696049.062327:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696049.062328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696049.062330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696049.062335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126040448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696049.062338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126040448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696049.062341:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696049.573507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696049.573610:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696050.597524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696050.981652:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298624367, at 4298624384 00000100:00100000:3.0:1787696051.621529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696052.645522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696053.669514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696054.181559:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696054.181569:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696054.181573:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696054.181582:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 3955 00000100:00080000:3.0:1787696054.181592:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696054.181596:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.181600:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.181604:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696054.181608:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696054.181611:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.181614:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.181617:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696054.181620:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696054.181634:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696054.181659:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126040576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696054.181673:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126040576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696054.181694:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696054.181698:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696054.181699:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696054.181703:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3955 00000100:00080000:3.0:1787696054.181719:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696054.181723:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.181726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.181728:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696054.181731:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696054.181737:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.181742:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.181747:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696054.181753:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696054.181762:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696054.181769:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696054.181784:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126040704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696054.181795:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126040704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696054.181813:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696054.181821:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696054.181824:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696054.181829:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3955 00000100:00080000:3.0:1787696054.181847:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696054.181850:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.181854:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.181859:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696054.181863:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696054.181867:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.181870:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.182082:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696054.182091:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696054.182100:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696054.182108:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696054.182124:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126040832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696054.182151:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126040832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696054.182164:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696054.182171:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696054.182174:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696054.182181:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3955 00000100:00080000:3.0:1787696054.182186:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696054.182192:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.182197:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.182199:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696054.182202:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696054.182207:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.182212:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.182216:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696054.182220:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696054.182224:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696054.182229:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696054.182239:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126040960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696054.182280:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126040960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696054.182287:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696054.182291:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696054.182294:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696054.182301:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 3955 00000100:00080000:3.0:1787696054.182306:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696054.182312:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.182315:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.182318:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696054.182319:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696054.182321:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696054.182323:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696054.182325:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696054.182327:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696054.182331:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696054.182336:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696054.182345:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126041088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696054.182371:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126041088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696054.182382:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (3965) 00000100:00100000:3.0:1787696054.693532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696055.717497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696056.101560:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298629487, at 4298629504 00000100:00100000:3.0:1787696056.741523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696057.765506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696058.789538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696059.813491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696060.837537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696061.221520:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298634607, at 4298634624 00000100:00100000:3.0:1787696061.861530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696062.885526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696063.909516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696064.933519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696065.957509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696066.341544:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298639727, at 4298639744 00000100:00100000:3.0:1787696066.981490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696068.005544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696068.133599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696069.029524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696069.157570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787696069.541793:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.541812:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126042496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696069.541845:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.541871:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696069.541911:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696069.541935:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.541987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696069/real 1787696069] req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696069.542082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696069.542091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696069.542098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126042496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696069.542111:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126042496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696069.542119:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696069.542132:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126042624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696069.542149:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542154:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696069.542164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696069.542169:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542176:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696069.542180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696069/real 1787696069] req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696069.542205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696069.542207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696069.542209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126042624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696069.542218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126042752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696069.542231:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542236:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696069.542243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696069.542248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542254:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696069.542257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696069/real 1787696069] req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542280:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696069.542285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696069.542287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696069.542289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126042752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696069.542297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126042880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696069.542309:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542314:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696069.542321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696069.542325:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542330:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696069.542334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696069/real 1787696069] req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696069.542359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696069.542361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696069.542364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126042880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696069.542371:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696069.542383:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542387:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696069.542394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696069.542398:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542403:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696069.542406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696069/real 1787696069] req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696069.542440:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696069.542446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696069.542448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696069.542451:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696069.542455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696069.542465:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126042624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696069.542472:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696069.542476:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126042752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696069.542481:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696069.542485:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126042880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696069.542489:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696069.542492:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126043008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696069.542496:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696070.053544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696070.181555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696071.077509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696071.077549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696071.205565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787696071.461549:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298644847, at 4298644864 00000100:00100000:3.0:1787696072.101613:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696072.229557:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696073.125516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696073.125595:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696073.253593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696074.149600:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696074.277550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787696074.661754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.661773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126043136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696074.661801:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.661812:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696074.661845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696074.661861:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.661891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696074/real 1787696074] req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.661907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.661918:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696074.661928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696074.661932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696074.661941:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.661956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126043136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696074.661964:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126043136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696074.661977:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696074.661997:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696074.662021:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662030:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696074.662043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696074.662051:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662064:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696074.662070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696074/real 1787696074] req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662083:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696074.662101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696074.662111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696074.662117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696074.662130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696074.662148:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662153:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696074.662163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696074.662168:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662174:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696074.662178:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696074/real 1787696074] req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662191:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696074.662196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696074.662198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696074.662200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696074.662214:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696074.662229:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662234:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696074.662241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696074.662245:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662252:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696074.662255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696074/real 1787696074] req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696074.662273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696074.662274:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696074.662277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696074.662290:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126043264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696074.662296:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696074.662300:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126043392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696074.662304:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696074.662308:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126043520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696074.662312:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696074.662318:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696074.662335:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662343:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696074.662356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696074.662363:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662373:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696074.662379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696074/real 1787696074] req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696074.662397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696074.662408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696074.662410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696074.662413:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696074.662439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696074.662444:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126043648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696074.662450:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696075.173504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696075.301624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787696076.197538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:3.0:1787696076.325546:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787696076.581536:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298649967, at 4298649984 00000100:00100000:3.0:1787696077.221542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696078.245544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696079.269606:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696079.781716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.781741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126043776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696079.781797:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.781826:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696079.781876:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696079.781897:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.781967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696079/real 1787696079] req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.781990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696079.782031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696079.782039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696079.782045:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126043776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696079.782074:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126043776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696079.782098:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696079.782141:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696079.782167:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782175:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696079.782188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696079.782195:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782217:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696079.782223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696079/real 1787696079] req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696079.782250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696079.782253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696079.782256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126043904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696079.782270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696079.782299:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782306:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696079.782318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696079.782325:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782334:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696079.782346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696079/real 1787696079] req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696079.782375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696079.782378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696079.782382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696079.782395:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696079.782415:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782458:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696079.782470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696079.782477:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782486:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696079.782491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696079/real 1787696079] req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696079.782539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696079.782542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696079.782546:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696079.782561:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696079.782580:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782587:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696079.782601:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696079.782608:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696104 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782616:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696079.782621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696079/real 1787696079] req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696079.782645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696079.782653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696079.782657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696079.782661:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696079.782668:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696079.782674:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126043904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696079.782683:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696079.782689:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126044032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696079.782695:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696079.782700:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126044160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696079.782708:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696079.782712:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126044288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696104 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696079.782719:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696080.293590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696081.317509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696081.701540:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298655087, at 4298655104 00000100:00100000:3.0:1787696082.341547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696083.365546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696084.389549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696084.901763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.901786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126044416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696084.901822:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.901834:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696084.901873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696084.901900:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.901943:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696084/real 1787696084] req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.901960:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.901974:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696084.901991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696084.901996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696084.902005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126044416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696084.902033:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126044416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696084.902073:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696084.902094:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696084.902123:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902149:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696084.902163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696084.902171:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902179:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696084.902183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696084/real 1787696084] req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696084.902220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696084.902224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696084.902228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696084.902252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696084.902268:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902274:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696084.902283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696084.902288:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902298:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696084.902304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696084/real 1787696084] req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696084.902336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696084.902340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696084.902344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696084.902381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902398:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696084.902415:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902448:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696084.902459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696084.902465:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902472:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696084.902476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696084/real 1787696084] req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902491:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696084.902496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696084.902498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696084.902500:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696084.902508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126044544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696084.902514:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696084.902524:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126044672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696084.902529:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696084.902532:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126044800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696084.902537:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696084.902542:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696084.902557:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902562:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696084.902570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696084.902574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696109 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902580:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696084.902583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696084/real 1787696084] req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696084.902604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696084.902610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696084.902612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696084.902619:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696084.902625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126044928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696084.902628:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126044928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696109 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696084.902634:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696085.413527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696086.437527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696086.821514:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298660207, at 4298660224 00000100:00100000:3.0:1787696087.461560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696088.485495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696088.485545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696089.509581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696089.509620:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696090.021761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.021778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126045056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696090.021821:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.021847:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696090.021895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696090.021922:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.021979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696090.022072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696090.022079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696090.022088:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126045056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696090.022106:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126045056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696090.022115:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696090.022131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696090.022149:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022155:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696090.022165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696090.022170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022178:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696090.022182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022204:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696090.022209:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696090.022211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696090.022214:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696090.022222:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696090.022235:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022240:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696090.022247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696090.022252:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022258:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696090.022279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696090.022311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696090.022313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696090.022316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696090.022325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696090.022340:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022345:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696090.022353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696090.022358:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022366:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696090.022372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696090.022439:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696090.022443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696090.022446:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022452:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696090.022456:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696090.022473:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022478:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696090.022487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696090.022493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022499:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696090.022503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696090/real 1787696090] req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696090.022522:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696090.022527:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696090.022530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696090.022532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696090.022537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696090.022540:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126045184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696090.022545:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696090.022550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126045312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696090.022554:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696090.022557:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126045440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696090.022561:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696090.022565:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126045568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696090.022570:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696090.533518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696091.557513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696091.941522:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298665327, at 4298665344 00000100:00100000:3.0:1787696092.581516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696093.605575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696094.629532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696095.141716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.141747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126045696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696095.141786:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.141815:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696095.141860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696095.141886:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.141944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.141967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.141979:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696095.141989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696095.141993:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696095.141998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126045696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696095.142014:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126045696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696095.142038:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696095.142055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696095.142081:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142091:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696095.142104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696095.142111:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142123:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696095.142129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696095.142163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696095.142167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696095.142184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696095.142212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696095.142237:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142245:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696095.142258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696095.142266:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142275:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696095.142282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696095.142322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696095.142326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696095.142329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126045952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696095.142345:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696095.142375:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142383:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696095.142396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696095.142403:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142412:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696095.142443:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696095.142475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696095.142478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696095.142483:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696095.142497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696095.142528:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142536:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696095.142549:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696095.142556:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142566:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696095.142573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696095/real 1787696095] req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696095.142593:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696095.142601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696095.142604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696095.142608:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696095.142615:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696095.142622:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126045824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696095.142630:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696095.142636:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126045952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696095.142655:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696095.142661:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126046080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696095.142669:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696095.142674:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126046208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696095.142681:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696095.653535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696096.677510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696097.061560:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298670447, at 4298670464 00000100:00100000:3.0:1787696097.701687:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696098.725504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696099.749605:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696100.261845:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.261889:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126046336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696100.261930:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.261951:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696100.262001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696100.262025:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262109:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696100/real 1787696100] req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262196:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696100.262223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696100.262228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696100.262236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126046336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696100.262273:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126046336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696100.262301:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696100.262324:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696100.262355:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262411:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696100.262467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696100.262495:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262548:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696100.262571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696100/real 1787696100] req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262663:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696100.262671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696100.262674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696100.262677:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696100.262689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696100.262707:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262712:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696100.262723:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696100.262728:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262735:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696100.262739:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696100/real 1787696100] req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262765:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262775:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696100.262784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696100.262787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696100.262790:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696100.262805:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126046464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696100.262814:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696100.262821:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126046592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696100.262828:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696100.262838:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696100.262863:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262870:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696100.262891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696100.262900:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262911:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696100.262917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696100/real 1787696100] req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262932:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696100.262940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696100.262944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696100.262948:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696100.262963:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.262972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696100.262987:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.262995:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696100.263007:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696100.263015:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.263036:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696100.263041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696100/real 1787696100] req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.263049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696100.263055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696100.263060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696100.263062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696100.263065:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696100.263069:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126046848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696100.263073:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126046720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696100.263079:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696100.263083:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126046848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696100.263105:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696100.773535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696101.797538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696102.181514:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298675567, at 4298675584 00000100:00100000:3.0:1787696102.821530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696103.845533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696104.869537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696105.381666:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.381691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126046976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696105.381728:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.381758:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696105.381798:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696105.381812:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.381838:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696105/real 1787696105] req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.381855:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.381875:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696105.381890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696105.381895:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696105.381902:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.381918:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126046976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696105.381928:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126046976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696105.381947:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696105.381973:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.381984:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696105.382023:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382046:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696105.382069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696105.382085:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382109:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696105.382120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696105/real 1787696105] req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696105.382142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696105.382144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696105.382147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696105.382156:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696105.382184:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382189:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696105.382198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696105.382203:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382210:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696105.382213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696105/real 1787696105] req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696105.382231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696105.382233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696105.382236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696105.382244:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126047104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696105.382249:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696105.382260:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126047232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696105.382266:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696105.382272:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696105.382287:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382292:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696105.382299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696105.382304:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382310:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696105.382313:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696105/real 1787696105] req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696105.382332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696105.382333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696105.382336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696105.382349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696105.382363:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382368:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696105.382375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696105.382379:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382384:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696105.382388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696105/real 1787696105] req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696105.382401:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696105.382405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696105.382407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696105.382409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696105.382414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126047488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696105.382439:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126047360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696105.382460:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696105.382467:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126047488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696105.382475:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696105.893524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696106.917558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696107.301517:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298680687, at 4298680704 00000100:00100000:3.0:1787696107.941567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696108.965623:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696109.989504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696111.013600:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696112.037499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696112.421534:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298685807, at 4298685824 00000100:00100000:3.0:1787696113.061642:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696114.085512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696115.109544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696116.133578:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696117.157528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696117.541486:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298690927, at 4298690944 00000100:00100000:3.0:1787696118.181485:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696119.205554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696120.229506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696121.253522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696122.277582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696122.661521:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298696047, at 4298696064 00000100:00100000:3.0:1787696123.301546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696124.325543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696125.349542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696126.373551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696127.397542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696127.781547:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298701167, at 4298701184 00000100:00100000:3.0:1787696128.421542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696128.934020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696125/real 1787696128] req@ffff9c5cc35af480 x1874531126049920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126049920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696128.934076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126049920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696128.934088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696128.934093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696128.934098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126049920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126049920:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787696128.934117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696125/real 1787696128] req@ffff9c5cc35adf80 x1874531126049792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126049792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696128.934143:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126049792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696128.934152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696128.934156:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696128.934182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126049792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126049792:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787696128.934200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696125/real 1787696128] req@ffff9c5cc35ac000 x1874531126049664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126049664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696128.934225:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126049664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696128.934233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696128.934237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696128.934241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126049664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126049664:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787696128.934258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696125/real 1787696128] req@ffff9c5cc35ad500 x1874531126049536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126049536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696128.934282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126049536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696128.934289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696128.934300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696128.934304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126049536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126049536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696128.934320:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126049920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696128.934335:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696128.934346:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126049792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696128.934353:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696128.934359:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126049664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696128.934367:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696128.934372:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126049536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696128.934379:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696128.934388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696125/real 1787696128] req@ffff9c5cc35af100 x1874531126050048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126050048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696128.934412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126050048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696128.934468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696128.934479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696128.934483:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126050048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696128.934577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696128.934585:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126050048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696150 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696128.934610:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696129.445532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696130.469522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696130.981745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.981771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126050176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696130.981808:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.981832:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696130.981881:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696130.981902:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.981944:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696130/real 1787696130] req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.981970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.981982:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696130.982016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696130.982032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696130.982039:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.982060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126050176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696130.982075:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126050176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696130.982096:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696130.982117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.982139:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696130.982167:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.982187:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696130.982211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696130.982225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.982248:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696130.982258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696130/real 1787696130] req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.982280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.982319:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696130.982328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696130.982332:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696130.982336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.982343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696130.982356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.982367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696130.982383:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.982389:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696130.982408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696130.982416:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983145:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696130.983154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696130/real 1787696130] req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696130.983201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696130.983206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696130.983211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696130.983227:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126050304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696130.983238:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696130.983246:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126050432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696130.983254:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696130.983266:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696130.983297:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983306:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696130.983325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696130.983341:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983352:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696130.983359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696130/real 1787696130] req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696130.983440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696130.983444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696130.983449:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696130.983465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696130.983488:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983494:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696130.983504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696130.983510:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696155 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983519:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696130.983525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696130/real 1787696130] req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983537:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696130.983568:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696130.983578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696130.983582:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696130.983587:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696130.983595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696130.983602:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126050560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696130.983612:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696130.983618:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126050688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696155 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696130.983627:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696131.493532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696132.517621:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696132.901555:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298706287, at 4298706304 00000100:00100000:3.0:1787696133.541529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696134.565560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696135.589531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696136.101814:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.101840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126050816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696136.101878:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.101911:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696136.101959:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696136.101989:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102079:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102096:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696136.102109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696136.102117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696136.102123:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102140:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126050816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696136.102157:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126050816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696136.102181:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696136.102208:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696136.102231:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102246:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696136.102257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696136.102263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102271:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787696136.102276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696136.102296:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696136.102298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696136.102300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126050944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696136.102318:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126051072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696136.102333:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102338:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696136.102351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696136.102357:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102363:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787696136.102367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696136.102387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696136.102389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696136.102391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126051072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787696136.102400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126051200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696136.102414:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102442:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696136.102450:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696136.102454:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102488:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787696136.102493:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696136.102513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696136.102516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696136.102520:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126051200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696136.102534:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102545:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126051328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696136.102559:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102568:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787696136.102581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696136.102589:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102599:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787696136.102613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696136/real 1787696136] req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696136.102632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696136.102637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696136.102639:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696136.102641:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696136.102653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126051328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787696136.102658:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126050944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696136.102664:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696136.102669:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126051072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696136.102674:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787696136.102677:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126051200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696136.102682:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787696136.102685:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126051328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696136.102689:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696136.613536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696137.637520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696137.637575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696138.021564:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298711407, at 4298711424 00000100:00100000:3.0:1787696138.661587:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696139.685556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696141.221664:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696141.221676:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696141.221681:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696141.221693:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4042 00000100:00080000:3.0:1787696141.221704:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696141.221707:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.221710:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.221713:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696141.221717:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696141.221719:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.221721:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.221724:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696141.221726:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696141.221740:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696141.221765:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126051456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696141.221783:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126051456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696141.221809:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696141.221812:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696141.221813:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696141.221831:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4042 00000100:00080000:3.0:1787696141.221837:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696141.221845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.221850:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.221856:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696141.221862:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696141.221873:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.221878:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.221880:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696141.221886:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696141.221904:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696141.221912:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696141.221924:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126051584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696141.221936:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126051584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696141.221986:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696141.221990:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696141.221992:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696141.221996:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4042 00000100:00080000:3.0:1787696141.222001:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696141.222004:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.222006:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.222008:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696141.222010:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696141.222012:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.222019:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.222021:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696141.222023:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696141.222028:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696141.222031:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696141.222043:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126051712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696141.222052:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126051712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696141.222059:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696141.222062:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696141.222065:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696141.222068:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4042 00000100:00080000:3.0:1787696141.222072:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696141.222077:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.222090:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.222101:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696141.222106:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696141.222109:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.222110:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.222112:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696141.222113:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696141.222124:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696141.222129:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696141.222134:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126051840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696141.222154:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126051840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696141.222161:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696141.222164:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696141.222165:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696141.222169:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4042 00000100:00080000:3.0:1787696141.222173:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696141.222176:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.222179:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.222182:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696141.222185:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696141.222191:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696141.222196:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696141.222203:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696141.222211:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696141.222218:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696141.222231:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696141.222249:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126051968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696141.222304:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126051968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696141.222331:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4052) 00000100:00000040:3.0:1787696143.141508:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298716527, at 4298716544 00000100:00000040:3.0:1787696148.261513:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298721647, at 4298721664 00000100:00100000:3.0:1787696149.925543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696150.949527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696151.461564:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696151.461573:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696151.461576:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696151.461584:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4052 00000100:00080000:3.0:1787696151.461593:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696151.461595:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.461599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.461601:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696151.461605:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696151.461607:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.461610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.461612:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696151.461616:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696151.461632:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696151.461666:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126052736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696151.461677:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126052736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696151.461703:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696151.461707:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696151.461709:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696151.461713:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4052 00000100:00080000:3.0:1787696151.461717:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696151.461722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.461737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.461739:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696151.461744:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696151.461750:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.461757:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.461761:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696151.461777:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696151.461790:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696151.461797:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696151.461811:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126052864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696151.461834:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126052864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696151.461847:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696151.461850:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696151.461851:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696151.461855:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4052 00000100:00080000:3.0:1787696151.461876:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696151.461881:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.461885:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.461887:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696151.461889:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696151.461891:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.461894:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.461897:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696151.461903:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696151.461919:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696151.461926:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696151.461943:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126052992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696151.461955:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126052992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696151.461974:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696151.461978:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696151.461981:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696151.461990:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4052 00000100:00080000:3.0:1787696151.461997:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696151.462003:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.462009:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.462013:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696151.462017:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696151.462025:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.462031:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.462036:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696151.462042:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696151.462048:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696151.462053:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696151.462065:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126053120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696151.462082:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126053120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696151.462120:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696151.462126:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696151.462128:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696151.462133:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4052 00000100:00080000:3.0:1787696151.462138:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696151.462145:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.462152:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.462155:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696151.462157:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696151.462164:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696151.462171:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696151.462176:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696151.462182:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696151.462188:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696151.462197:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696151.462227:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126053248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696151.462246:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126053248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696151.462263:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4062) 00000100:00100000:3.0:1787696151.973510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696151.973581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696152.997523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696153.381523:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298726767, at 4298726784 00000100:00000040:3.0:1787696158.501486:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298731887, at 4298731904 00000100:00100000:3.0:1787696159.141532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696160.165525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696161.189547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696161.701546:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696161.701556:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696161.701560:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696161.701570:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4062 00000100:00080000:3.0:1787696161.701580:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696161.701584:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701588:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701591:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696161.701596:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696161.701599:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701602:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701605:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696161.701608:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701620:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696161.701645:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126054016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696161.701659:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126054016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696161.701680:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696161.701684:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696161.701686:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696161.701690:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4062 00000100:00080000:3.0:1787696161.701694:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696161.701711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701716:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696161.701719:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696161.701723:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701731:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696161.701734:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696161.701744:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696161.701753:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696161.701766:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126054144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696161.701777:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126054144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696161.701787:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696161.701791:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696161.701796:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696161.701805:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4062 00000100:00080000:3.0:1787696161.701813:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696161.701818:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701821:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701823:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696161.701825:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696161.701828:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701831:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701833:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696161.701841:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696161.701847:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696161.701851:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696161.701865:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126054272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696161.701882:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126054272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696161.701894:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696161.701897:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696161.701899:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696161.701903:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4062 00000100:00080000:3.0:1787696161.701907:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696161.701911:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701917:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701923:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696161.701928:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696161.701932:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.701939:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.701945:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696161.701951:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696161.701958:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696161.701965:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696161.701979:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126054400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696161.702004:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126054400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696161.702021:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696161.702042:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696161.702047:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696161.702054:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4062 00000100:00080000:3.0:1787696161.702062:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696161.702066:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.702072:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.702077:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696161.702083:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696161.702090:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696161.702093:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696161.702097:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696161.702104:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696161.702110:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696161.702115:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696161.702128:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126054528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696161.702160:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126054528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696161.702186:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4072) 00000100:00100000:3.0:1787696162.213533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696163.237536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696163.621526:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298737007, at 4298737024 00000100:00100000:3.0:1787696164.261518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696165.285552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696166.309527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696167.333520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696168.357498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696168.741511:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298742127, at 4298742144 00000100:00100000:3.0:1787696169.381510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696170.405483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696171.429526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696172.453510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696173.477518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696173.861501:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298747247, at 4298747264 00000100:00100000:3.0:1787696174.501533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696175.525606:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696176.549527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696177.573622:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696178.597530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696178.981479:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298752367, at 4298752384 00000100:00100000:3.0:1787696179.621547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696180.645520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696181.669525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696182.693545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696183.717514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696184.101527:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298757487, at 4298757504 00000100:00100000:3.0:1787696184.741538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696185.765502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696186.789522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696187.813494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696188.837540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696189.221505:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298762607, at 4298762624 00000100:00100000:3.0:1787696189.861561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696190.885538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696191.909523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696192.933540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696193.957514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696194.341544:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298767727, at 4298767744 00000100:00100000:3.0:1787696194.981545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000400:00000100:3.0:1787696195.493715:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787696195.493754:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696195.493768:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696195.493796:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126057856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696195.493811:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696195.493817:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126057984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696195.493824:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696195.493831:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126058112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696195.493838:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696195.493844:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126058240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696195.493853:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696195.493859:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126058368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696196.005556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696197.029523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696198.053540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696199.077549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696199.461560:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298772847, at 4298772864 00000100:00100000:3.0:1787696200.101562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696201.125607:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696202.149532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696203.173532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696204.197603:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696204.581541:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298777967, at 4298777984 00000100:00100000:3.0:1787696205.221501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696206.245570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696207.269519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696208.293544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696209.317518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696209.701537:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298783087, at 4298783104 00000100:00100000:3.0:1787696210.341532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696211.365517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696212.389499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696213.413563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696214.437522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696214.821528:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298788207, at 4298788224 00000100:00100000:3.0:1787696215.461547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696216.485545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696217.509509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696218.021578:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696218.021589:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696218.021594:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696218.021607:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4119 00000100:00080000:3.0:1787696218.021620:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696218.021624:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021629:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021634:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696218.021641:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696218.021646:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021677:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021681:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696218.021684:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021707:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696218.021737:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126061056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696218.021751:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126061056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696218.021771:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696218.021774:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696218.021776:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696218.021780:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4119 00000100:00080000:3.0:1787696218.021783:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696218.021786:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021789:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021791:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696218.021794:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696218.021796:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021800:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696218.021805:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696218.021815:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696218.021823:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696218.021834:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126061184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696218.021843:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126061184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696218.021856:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696218.021859:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696218.021860:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696218.021866:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4119 00000100:00080000:3.0:1787696218.021873:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696218.021879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021887:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021889:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696218.021891:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696218.021893:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021902:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021904:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696218.021906:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696218.021911:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696218.021915:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696218.021923:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126061312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696218.021930:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126061312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696218.021943:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696218.021945:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696218.021947:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696218.021951:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4119 00000100:00080000:3.0:1787696218.021955:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696218.021967:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021972:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021974:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696218.021976:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696218.021979:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.021988:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.021996:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696218.021999:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696218.022006:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696218.022012:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696218.022025:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126061440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696218.022067:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126061440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696218.022093:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696218.022096:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696218.022098:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696218.022102:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4119 00000100:00080000:3.0:1787696218.022107:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696218.022115:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.022121:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.022124:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696218.022127:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696218.022135:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696218.022141:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696218.022144:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696218.022157:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696218.022168:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696218.022173:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696218.022186:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126061568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696218.022198:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126061568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696218.022206:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4129) 00000100:00100000:3.0:1787696218.533539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696219.557557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696219.941554:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298793327, at 4298793344 00000100:00100000:3.0:1787696220.581526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696221.605573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696222.629516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696225.061496:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298798447, at 4298798464 00000100:00100000:3.0:1787696226.725585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696227.749527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696228.773555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696229.797509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696230.181493:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298803567, at 4298803584 00000100:00100000:3.0:1787696230.821503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696231.845511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696232.869504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696233.893528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696234.917500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696235.301516:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298808687, at 4298808704 00000100:00100000:3.0:1787696235.941511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696236.965535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696237.989546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696239.013508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696240.037522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696240.421555:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298813807, at 4298813824 00000100:00100000:3.0:1787696241.061576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696242.085523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696243.109539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696244.133498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696245.157523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696245.541528:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298818927, at 4298818944 00000100:00100000:3.0:1787696246.181508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696247.205514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696248.229519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696249.253535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696250.277517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696250.661525:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298824047, at 4298824064 00000100:00100000:3.0:1787696251.301528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696252.325515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696253.349529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696254.373528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696255.397529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696255.781496:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298829167, at 4298829184 00000100:00100000:3.0:1787696256.421609:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696257.445521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696258.469523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696259.493858:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696260.517537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696260.901525:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298834287, at 4298834304 00000100:00100000:3.0:1787696261.541503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696262.565568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696263.589493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696264.613594:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696265.637535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696266.021608:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298839407, at 4298839424 00000100:00080000:3.0:1787696269.221574:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696269.221582:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696269.221586:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696269.221595:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4170 00000100:00080000:3.0:1787696269.221604:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696269.221606:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221613:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696269.221617:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696269.221619:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221621:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221624:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696269.221626:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221638:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696269.221658:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126067456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696269.221670:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126067456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696269.221698:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696269.221701:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696269.221702:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696269.221706:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4170 00000100:00080000:3.0:1787696269.221708:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696269.221724:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221727:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221729:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696269.221732:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696269.221734:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221737:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221739:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696269.221744:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696269.221751:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696269.221758:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696269.221778:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126067584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696269.221793:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126067584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696269.221804:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696269.221808:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696269.221810:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696269.221818:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4170 00000100:00080000:3.0:1787696269.221827:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696269.221834:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221839:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221841:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696269.221843:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696269.221847:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221851:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221853:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696269.221871:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696269.221880:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696269.221886:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696269.221897:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126067712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696269.221920:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126067712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696269.221930:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696269.221934:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696269.221936:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696269.221944:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4170 00000100:00080000:3.0:1787696269.221953:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696269.221960:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221969:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221974:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696269.221977:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696269.221986:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.221990:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.221994:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696269.222002:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696269.222009:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696269.222015:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696269.222026:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126067840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696269.222041:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126067840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696269.222072:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696269.222074:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696269.222076:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696269.222079:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4170 00000100:00080000:3.0:1787696269.222085:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696269.222092:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.222099:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.222105:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696269.222108:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696269.222110:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696269.222112:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696269.222114:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696269.222118:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696269.222126:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696269.222133:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696269.222144:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126067968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696269.222159:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126067968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696269.222190:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4180) 00000100:00000040:3.0:1787696271.141490:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298844527, at 4298844544 00000100:00100000:3.0:1787696272.805568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696273.829500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696274.341566:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696274.341581:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696274.341613:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696274.341628:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4175 00000100:00080000:3.0:1787696274.341641:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696274.341645:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.341649:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341652:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696274.341658:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696274.341660:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.341662:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341665:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696274.341667:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341682:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696274.341711:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126068096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696274.341724:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126068096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696274.341744:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696274.341747:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696274.341748:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696274.341754:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4175 00000100:00080000:3.0:1787696274.341759:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696274.341762:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.341766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341769:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696274.341773:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696274.341777:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.341783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341794:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696274.341802:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696274.341817:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696274.341823:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696274.341835:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126068224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696274.341844:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126068224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696274.341852:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696274.341855:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696274.341857:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696274.341860:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4175 00000100:00080000:3.0:1787696274.341865:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696274.341870:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.341873:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341876:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696274.341880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696274.341887:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.341893:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341897:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696274.341904:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696274.341910:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696274.341918:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696274.341930:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126068352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696274.341959:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126068352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696274.341966:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696274.341969:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696274.341971:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696274.341975:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4175 00000100:00080000:3.0:1787696274.341979:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696274.341984:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.341990:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.341994:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696274.341998:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696274.342003:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.342005:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.342007:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696274.342009:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696274.342014:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696274.342018:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696274.342027:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126068480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696274.342040:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126068480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696274.342066:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696274.342073:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696274.342076:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696274.342103:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4175 00000100:00080000:3.0:1787696274.342109:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696274.342115:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.342120:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.342123:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696274.342127:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696274.342135:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696274.342141:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696274.342146:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696274.342149:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696274.342155:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696274.342160:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696274.342174:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126068608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696274.342192:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126068608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696274.342210:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4185) 00000100:00100000:3.0:1787696274.853564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696275.877512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696276.261515:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298849647, at 4298849664 00000100:00100000:3.0:1787696276.901586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696277.925537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696278.949512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696279.973529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696280.997498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696281.381492:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298854767, at 4298854784 00000100:00100000:3.0:1787696282.021513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696283.045518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696284.069520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696285.093524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696286.117521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696286.501527:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298859887, at 4298859904 00000100:00100000:3.0:1787696287.141510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696289.701523:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696289.701530:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696289.701534:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696289.701544:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4190 00000100:00080000:3.0:1787696289.701553:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696289.701555:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701559:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701561:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696289.701565:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696289.701568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701569:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701572:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696289.701574:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701586:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696289.701607:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126070016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696289.701625:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126070016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696289.701641:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696289.701644:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696289.701645:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696289.701669:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4190 00000100:00080000:3.0:1787696289.701674:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696289.701678:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701684:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701690:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696289.701695:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696289.701701:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701708:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701715:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696289.701717:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696289.701725:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696289.701730:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696289.701745:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126070144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696289.701764:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126070144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696289.701776:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696289.701780:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696289.701782:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696289.701790:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4190 00000100:00080000:3.0:1787696289.701794:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696289.701797:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701800:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696289.701802:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696289.701804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701813:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701815:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696289.701845:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696289.701850:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696289.701855:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696289.701868:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126070272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696289.701876:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126070272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696289.701883:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696289.701886:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696289.701887:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696289.701893:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4190 00000100:00080000:3.0:1787696289.701900:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696289.701904:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701908:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701909:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696289.701913:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696289.701917:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.701919:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.701921:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696289.701928:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696289.701937:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696289.701947:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696289.701962:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126070400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696289.701984:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126070400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696289.702004:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696289.702009:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696289.702010:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696289.702015:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4190 00000100:00080000:3.0:1787696289.702021:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696289.702024:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.702026:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.702027:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696289.702029:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696289.702032:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696289.702037:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696289.702041:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696289.702046:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696289.702052:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696289.702057:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696289.702067:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126070528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696289.702075:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126070528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696289.702088:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4200) 00000100:00000040:3.0:1787696291.621516:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298865007, at 4298865024 00000100:00080000:3.0:1787696294.821523:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696294.821546:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696294.821549:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696294.821558:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4195 00000100:00080000:3.0:1787696294.821567:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696294.821570:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821574:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821577:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696294.821581:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696294.821584:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821586:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821588:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696294.821591:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821601:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696294.821624:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126070656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696294.821641:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126070656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696294.821665:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696294.821669:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696294.821671:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696294.821676:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4195 00000100:00080000:3.0:1787696294.821681:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696294.821685:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821688:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821691:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696294.821694:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696294.821709:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821717:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821724:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696294.821727:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696294.821751:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696294.821756:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696294.821768:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126070784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696294.821777:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126070784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696294.821785:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696294.821787:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696294.821789:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696294.821794:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4195 00000100:00080000:3.0:1787696294.821799:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696294.821805:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821815:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696294.821819:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696294.821827:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821833:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821835:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696294.821838:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696294.821843:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696294.821849:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696294.821862:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126070912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696294.821884:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126070912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696294.821906:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696294.821910:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696294.821912:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696294.821916:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4195 00000100:00080000:3.0:1787696294.821920:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696294.821923:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821925:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821927:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696294.821930:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696294.821936:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.821942:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.821948:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696294.821955:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696294.821961:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696294.821970:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696294.821995:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126071040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696294.822021:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126071040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696294.822039:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696294.822044:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696294.822046:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696294.822060:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4195 00000100:00080000:3.0:1787696294.822067:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696294.822071:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.822074:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.822076:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696294.822080:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696294.822086:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696294.822092:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696294.822099:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696294.822106:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696294.822114:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696294.822121:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696294.822136:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126071168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696294.822157:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126071168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696294.822172:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4205) 00000100:00100000:3.0:1787696295.333536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696296.357571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696296.741523:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298870127, at 4298870144 00000100:00000040:3.0:1787696301.861535:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298875247, at 4298875264 00000100:00100000:3.0:1787696302.501581:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696303.525533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696304.549545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696305.573582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696306.597572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696306.981546:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298880367, at 4298880384 00000100:00100000:3.0:1787696307.621585:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696308.645514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696309.669530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696310.693535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696311.717538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696312.101551:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298885487, at 4298885504 00000100:00100000:3.0:1787696312.741500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696313.765554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696314.789496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696315.813518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696316.837527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696317.221507:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298890607, at 4298890624 00000100:00100000:3.0:1787696317.861521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696318.885536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696319.909505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696320.933515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696321.957486:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696322.341471:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298895727, at 4298895744 00000100:00100000:3.0:1787696322.981520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696324.005532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696325.029493:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696326.053530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696327.077521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696327.461515:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298900847, at 4298900864 00000100:00100000:3.0:1787696328.101512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696329.125560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696330.149525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696332.581863:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298905967, at 4298905984 00000100:00000040:3.0:1787696337.701485:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298911087, at 4298911104 00000100:00100000:3.0:1787696339.365558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696340.389522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696341.413517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696342.437547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696342.821522:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298916207, at 4298916224 00000100:00100000:3.0:1787696343.461548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696344.485514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696347.941559:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298921327, at 4298921344 00000100:00080000:3.0:1787696351.141543:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696351.141551:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696351.141555:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696351.141564:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4252 00000100:00080000:3.0:1787696351.141574:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696351.141576:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141580:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141582:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696351.141587:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696351.141589:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141591:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141594:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696351.141596:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141608:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696351.141630:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126077696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696351.141642:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126077696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696351.141661:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696351.141663:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696351.141677:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696351.141681:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4252 00000100:00080000:3.0:1787696351.141684:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696351.141687:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141690:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141692:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696351.141694:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696351.141696:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141704:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141706:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696351.141710:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696351.141719:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696351.141726:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696351.141738:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126077824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696351.141755:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126077824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696351.141776:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696351.141778:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696351.141780:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696351.141782:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4252 00000100:00080000:3.0:1787696351.141785:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696351.141787:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141789:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141790:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696351.141792:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696351.141800:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141802:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141804:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696351.141809:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696351.141816:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696351.141822:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696351.141834:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126077952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696351.141846:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126077952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696351.141852:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696351.141855:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696351.141856:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696351.141858:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4252 00000100:00080000:3.0:1787696351.141861:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696351.141864:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141866:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141868:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696351.141870:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696351.141871:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141873:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141874:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696351.141879:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696351.141885:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696351.141890:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696351.141896:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126078080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696351.141929:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126078080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696351.141936:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696351.141939:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696351.141940:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696351.141943:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4252 00000100:00080000:3.0:1787696351.141946:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696351.141949:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141951:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141953:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696351.141957:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696351.141964:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696351.141969:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696351.141974:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696351.141978:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696351.141985:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696351.141989:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696351.142000:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126078208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696351.142027:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126078208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696351.142050:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4262) 00000100:00100000:3.0:1787696352.677590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696353.061561:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298926447, at 4298926464 00000100:00100000:3.0:1787696353.701510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696354.725597:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696355.749525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696356.773542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696357.797539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696358.181552:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298931567, at 4298931584 00000100:00100000:3.0:1787696358.821554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696359.845547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696360.869550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696361.381554:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696361.381570:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696361.381574:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696361.381596:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4262 00000100:00080000:3.0:1787696361.381607:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696361.381611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.381616:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.381620:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696361.381624:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696361.381628:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.381631:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.381634:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696361.381638:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696361.381654:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696361.381683:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126078976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696361.381699:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126078976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696361.381723:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696361.381727:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696361.381744:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696361.381750:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4262 00000100:00080000:3.0:1787696361.381756:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696361.381760:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.381765:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.381768:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696361.381772:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696361.381776:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.381783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.381788:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696361.381795:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696361.381810:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696361.381819:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696361.381837:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126079104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696361.381855:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126079104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696361.381876:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696361.381881:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696361.381883:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696361.381889:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4262 00000100:00080000:3.0:1787696361.381894:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696361.381898:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.381904:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.381910:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696361.381915:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696361.381929:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.381937:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.381943:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696361.381948:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696361.381954:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696361.381960:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696361.381973:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126079232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696361.381995:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126079232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696361.382011:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696361.382021:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696361.382025:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696361.382033:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4262 00000100:00080000:3.0:1787696361.382044:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696361.382051:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.382061:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.382068:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696361.382072:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696361.382078:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.382087:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.382094:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696361.382101:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696361.382110:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696361.382118:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696361.382136:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126079360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696361.382163:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126079360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696361.382183:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696361.382188:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696361.382191:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696361.382197:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4262 00000100:00080000:3.0:1787696361.382205:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696361.382215:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.382222:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.382225:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696361.382228:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696361.382232:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696361.382240:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696361.382244:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696361.382248:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696361.382257:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696361.382267:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696361.382282:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126079488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696361.382299:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126079488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696361.382324:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4272) 00000100:00100000:3.0:1787696361.893540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696362.917529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696363.301524:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298936687, at 4298936704 00000100:00100000:3.0:1787696363.941586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696364.965505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696365.989559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696368.421604:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298941807, at 4298941824 00000100:00000040:3.0:1787696373.541550:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298946927, at 4298946944 00000100:00100000:3.0:1787696374.181590:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696375.205480:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696376.229551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696377.253526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696378.277508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696378.661498:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298952047, at 4298952064 00000100:00100000:3.0:1787696379.301530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696380.325516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696381.349575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696382.373524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696383.397522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696383.781501:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298957167, at 4298957184 00000100:00100000:3.0:1787696384.421539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696385.445560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696386.469530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696387.493527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696388.517558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696388.901533:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298962287, at 4298962304 00000100:00100000:3.0:1787696389.541518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696390.565532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696391.589524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696392.613508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696392.613594:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696393.637529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696394.021510:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298967407, at 4298967424 00000100:00100000:3.0:1787696394.661547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696395.685534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696396.709528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696397.733526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696398.757548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696399.141653:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298972527, at 4298972544 00000100:00100000:3.0:1787696399.781537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696400.805576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696401.829497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696402.853536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696403.877502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696404.261533:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298977647, at 4298977664 00000100:00100000:3.0:1787696404.901525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696405.925477:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696406.949553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696407.461523:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696407.461532:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696407.461535:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696407.461544:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4308 00000100:00080000:3.0:1787696407.461553:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696407.461556:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461559:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461562:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696407.461566:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696407.461569:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461570:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461573:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696407.461575:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461595:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696407.461639:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126084736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696407.461656:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126084736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696407.461680:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696407.461684:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696407.461686:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696407.461691:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4308 00000100:00080000:3.0:1787696407.461696:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696407.461700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461703:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461704:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696407.461707:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696407.461710:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461712:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461715:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696407.461720:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696407.461733:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696407.461739:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696407.461754:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126084864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696407.461763:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126084864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696407.461773:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696407.461781:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696407.461806:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696407.461814:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4308 00000100:00080000:3.0:1787696407.461819:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696407.461823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461827:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461830:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696407.461833:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696407.461838:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461846:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461848:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696407.461849:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696407.461854:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696407.461860:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696407.461870:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126084992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696407.461892:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126084992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696407.461904:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696407.461908:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696407.461910:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696407.461915:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4308 00000100:00080000:3.0:1787696407.461919:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696407.461923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461927:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461930:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696407.461935:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696407.461952:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.461959:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.461966:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696407.461971:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696407.461979:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696407.461987:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696407.462004:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126085120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696407.462027:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126085120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696407.462052:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696407.462059:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696407.462061:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696407.462067:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4308 00000100:00080000:3.0:1787696407.462074:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696407.462077:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.462081:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.462085:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696407.462087:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696407.462089:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696407.462091:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696407.462092:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696407.462094:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696407.462098:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696407.462101:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696407.462111:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126085248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696407.462135:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126085248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696407.462146:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4318) 00000100:00100000:3.0:1787696407.973507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696408.997528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696409.381498:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298982767, at 4298982784 00000100:00100000:3.0:1787696414.117543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696414.501506:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298987887, at 4298987904 00000100:00100000:3.0:1787696415.141608:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696416.165718:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696417.189508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696418.213531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696419.237495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696419.621495:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298993007, at 4298993024 00000100:00080000:3.0:1787696422.821569:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696422.821580:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696422.821584:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696422.821596:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4323 00000100:00080000:3.0:1787696422.821609:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696422.821611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.821615:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821618:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696422.821623:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696422.821626:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.821628:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821630:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696422.821632:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821671:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696422.821701:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126086656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696422.821715:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126086656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696422.821738:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696422.821741:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696422.821743:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696422.821746:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4323 00000100:00080000:3.0:1787696422.821750:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696422.821752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.821754:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821757:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696422.821760:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696422.821762:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.821764:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821766:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696422.821768:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696422.821777:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696422.821781:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696422.821792:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126086784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696422.821805:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126086784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696422.821859:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696422.821870:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696422.821873:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696422.821888:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4323 00000100:00080000:3.0:1787696422.821891:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696422.821894:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.821896:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821899:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696422.821901:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696422.821907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.821909:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821911:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696422.821913:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696422.821917:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696422.821927:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696422.821947:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126086912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696422.821955:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126086912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696422.821961:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696422.821965:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696422.821967:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696422.821969:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4323 00000100:00080000:3.0:1787696422.821972:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696422.821976:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.821989:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.821998:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696422.822001:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696422.822004:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.822006:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.822011:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696422.822016:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696422.822022:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696422.822029:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696422.822046:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126087040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696422.822055:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126087040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696422.822062:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696422.822064:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696422.822066:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696422.822071:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4323 00000100:00080000:3.0:1787696422.822078:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696422.822082:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.822084:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.822085:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696422.822086:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696422.822088:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696422.822092:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696422.822100:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696422.822111:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696422.822125:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696422.822138:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696422.822155:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126087168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696422.822170:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126087168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696422.822243:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4333) 00000100:00100000:3.0:1787696424.357566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696424.741514:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4298998127, at 4298998144 00000100:00100000:3.0:1787696425.381520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696426.405536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696427.429493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696428.453543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696429.477514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696429.861526:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299003247, at 4299003264 00000100:00000040:3.0:1787696434.981498:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299008367, at 4299008384 00000100:00100000:3.0:1787696438.693565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696439.717563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696440.101548:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299013487, at 4299013504 00000100:00100000:3.0:1787696440.741542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696441.765519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696442.789524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787696443.301517:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696443.301535:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696443.301544:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4344 00000100:00080000:0.0:1787696443.301553:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696443.301556:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301559:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301562:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696443.301566:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696443.301569:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301571:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301573:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696443.301576:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301584:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696443.301611:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126089216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696443.301629:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126089216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696443.301657:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696443.301661:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696443.301663:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696443.301669:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4344 00000100:00080000:0.0:1787696443.301674:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696443.301678:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301682:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301685:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696443.301688:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696443.301692:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301697:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301708:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696443.301714:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696443.301724:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696443.301729:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696443.301741:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126089344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696443.301753:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126089344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696443.301761:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696443.301769:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696443.301771:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696443.301777:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4344 00000100:00080000:0.0:1787696443.301782:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696443.301785:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301788:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301789:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696443.301792:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696443.301794:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301795:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301797:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696443.301798:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696443.301802:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696443.301806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696443.301814:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126089472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696443.301841:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126089472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696443.301848:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696443.301851:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696443.301852:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696443.301856:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4344 00000100:00080000:0.0:1787696443.301859:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696443.301864:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301870:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301874:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696443.301878:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696443.301882:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.301886:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696443.301890:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696443.301897:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696443.301904:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696443.301915:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126089600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696443.301932:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126089600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696443.301957:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696443.301961:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696443.301964:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696443.301977:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4344 00000100:00080000:0.0:1787696443.301983:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696443.301993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.301999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.302004:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696443.302008:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696443.302021:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696443.302034:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696443.302038:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696443.302044:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696443.302049:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696443.302054:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696443.302065:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126089728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696443.302096:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126089728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696443.302106:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4354) 00000100:00100000:3.0:1787696443.813519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696444.837514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696445.221490:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299018607, at 4299018624 00000100:00100000:3.0:1787696445.861512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696446.885502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696447.909520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696448.421541:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696448.421580:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696448.421585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696448.421596:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4349 00000100:00080000:3.0:1787696448.421610:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696448.421613:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421616:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421620:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696448.421624:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696448.421627:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421629:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696448.421634:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421646:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696448.421682:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126089856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696448.421696:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126089856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696448.421720:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696448.421723:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696448.421725:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696448.421728:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4349 00000100:00080000:3.0:1787696448.421731:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696448.421732:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421734:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421736:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696448.421738:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696448.421747:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421750:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421752:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696448.421755:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696448.421764:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696448.421769:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696448.421781:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126089984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696448.421795:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126089984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696448.421809:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696448.421811:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696448.421813:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696448.421816:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4349 00000100:00080000:3.0:1787696448.421820:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696448.421822:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421824:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421825:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696448.421827:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696448.421829:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421832:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421837:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696448.421843:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696448.421850:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696448.421856:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696448.421866:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126090112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696448.421877:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126090112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696448.421884:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696448.421886:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696448.421887:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696448.421898:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4349 00000100:00080000:3.0:1787696448.421905:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696448.421911:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421918:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696448.421923:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696448.421928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.421931:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.421937:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696448.421944:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696448.421950:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696448.421954:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696448.421964:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126090240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696448.421972:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126090240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696448.421980:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696448.421982:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696448.421983:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696448.421995:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4349 00000100:00080000:3.0:1787696448.422000:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696448.422006:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.422015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.422022:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696448.422027:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696448.422036:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696448.422042:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696448.422049:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696448.422056:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696448.422065:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696448.422071:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696448.422098:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126090368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696448.422111:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126090368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696448.422122:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4359) 00000100:00100000:3.0:1787696448.933490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696449.957511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696450.341520:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299023727, at 4299023744 00000100:00100000:3.0:1787696450.981478:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696452.005555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696453.029560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696453.029574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696454.053506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696454.053508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696455.077498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696455.077523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696455.461527:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299028847, at 4299028864 00000100:00100000:0.0:1787696456.101488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696456.101489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696457.125571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696457.125583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696458.149481:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696458.149491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696459.173514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696459.173556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696460.197510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696460.197510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696460.581573:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299033967, at 4299033984 00000100:00100000:0.0:1787696461.221520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696461.221520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000400:00000100:3.0:1787696461.733763:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787696461.733823:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696461.733845:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696461.733925:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126091136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696483 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696461.733949:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696461.733959:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126091264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696483 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696461.733971:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696461.733981:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126091392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696483 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696461.733991:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696461.734000:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126091520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696483 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696461.734010:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696461.734020:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126091648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696483 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696462.245595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696462.245595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696463.269536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696463.269545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696463.781580:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696463.781588:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696463.781603:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696463.781612:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4364 00000100:00080000:3.0:1787696463.781620:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696463.781623:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.781627:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.781630:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696463.781634:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696463.781636:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.781638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.781640:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696463.781643:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696463.781653:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696463.781675:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126091776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696463.781688:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126091776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696463.781720:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696463.781725:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696463.781727:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696463.781732:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4364 00000100:00080000:3.0:1787696463.781736:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696463.781739:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.781742:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.781745:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696463.781748:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696463.781752:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.781755:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.781765:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696463.781769:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696463.781781:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696463.781787:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696463.781805:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126091904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696463.781866:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126091904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696463.781895:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696463.781899:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696463.781901:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696463.781907:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4364 00000100:00080000:3.0:1787696463.781912:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696463.781916:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.781929:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.781937:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696463.781945:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696463.781963:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.781976:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.781980:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696463.781983:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696463.781991:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696463.781999:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696463.782011:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126092032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696463.782054:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126092032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696463.782068:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696463.782087:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696463.782094:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696463.782104:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4364 00000100:00080000:3.0:1787696463.782112:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696463.782117:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.782120:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.782124:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696463.782127:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696463.782132:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.782136:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.782140:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696463.782150:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696463.782176:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696463.782189:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696463.782212:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126092160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696463.782263:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126092160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696463.782281:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696463.782293:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696463.782296:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696463.782324:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4364 00000100:00080000:3.0:1787696463.782335:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696463.782344:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.782355:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.782365:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696463.782372:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696463.782378:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696463.782386:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696463.782390:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696463.782393:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696463.782402:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696463.782413:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696463.782578:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126092288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696463.782600:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126092288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696463.782631:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4374) 00000100:00100000:0.0:1787696464.293562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696464.293562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696465.317543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696465.317552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696465.701967:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299039087, at 4299039104 00000100:00100000:3.0:1787696466.341508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696466.341512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696467.365540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696467.365557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696468.389500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696468.389528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696469.413498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696469.413500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696470.437492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696470.437506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696470.821524:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299044207, at 4299044224 00000100:00100000:0.0:1787696471.461493:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696471.461493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696472.485480:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696472.485487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696473.509520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696473.509520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696474.021593:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696474.021604:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696474.021610:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696474.021621:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4375 00000100:00080000:3.0:1787696474.021632:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696474.021637:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021657:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021661:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696474.021667:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696474.021671:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021674:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021677:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696474.021680:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021695:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696474.021723:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126093056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696474.021741:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126093056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696474.021764:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696474.021768:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696474.021770:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696474.021775:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4375 00000100:00080000:3.0:1787696474.021779:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696474.021782:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021800:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021803:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696474.021807:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696474.021811:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021814:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021817:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696474.021820:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696474.021831:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696474.021840:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696474.021855:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126093184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696474.021865:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126093184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696474.021872:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696474.021875:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696474.021876:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696474.021879:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4375 00000100:00080000:3.0:1787696474.021883:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696474.021888:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021894:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021899:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696474.021900:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696474.021902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021904:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021905:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696474.021909:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696474.021922:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696474.021927:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696474.021935:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126093312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696474.021942:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126093312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696474.021948:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696474.021950:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696474.021951:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696474.021956:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4375 00000100:00080000:3.0:1787696474.021961:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696474.021965:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021968:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021969:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696474.021971:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696474.021973:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.021976:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.021977:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696474.021982:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696474.021987:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696474.021990:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696474.021996:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126093440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696474.022016:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126093440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696474.022031:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696474.022034:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696474.022035:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696474.022038:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4375 00000100:00080000:3.0:1787696474.022041:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696474.022044:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.022049:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.022052:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696474.022055:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696474.022061:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696474.022067:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696474.022073:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696474.022077:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696474.022084:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696474.022094:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696474.022109:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126093568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696474.022130:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126093568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696474.022163:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4385) 00000100:00100000:3.0:1787696474.533561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696474.533572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696475.557522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696475.557561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696475.941500:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299049327, at 4299049344 00000100:00100000:0.0:1787696476.581534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696476.581536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696477.605500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696477.605504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696478.629532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696478.629532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696479.141549:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696479.141576:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696479.141581:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696479.141593:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4380 00000100:00080000:3.0:1787696479.141606:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696479.141608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.141612:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.141615:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696479.141621:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696479.141623:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.141625:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.141627:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696479.141630:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696479.141645:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696479.141678:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126093696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696479.141691:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126093696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696479.141710:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696479.141713:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696479.141715:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696479.141718:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4380 00000100:00080000:3.0:1787696479.141720:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696479.141722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.141723:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.141745:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696479.141747:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696479.141751:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.141754:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.141757:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696479.141760:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696479.141772:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696479.141778:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696479.141792:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126093824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696479.141813:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126093824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696479.141839:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696479.141850:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696479.141853:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696479.141862:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4380 00000100:00080000:3.0:1787696479.141867:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696479.141874:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.141882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.141885:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696479.141887:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696479.141888:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.141890:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.141892:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696479.141895:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696479.141900:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696479.141909:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696479.141920:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126093952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696479.141970:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126093952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696479.141992:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696479.141995:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696479.141997:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696479.142001:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4380 00000100:00080000:3.0:1787696479.142011:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696479.142015:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.142017:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.142018:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696479.142020:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696479.142028:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.142033:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.142038:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696479.142041:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696479.142048:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696479.142051:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696479.142063:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126094080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696479.142103:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126094080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696479.142121:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696479.142132:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696479.142134:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696479.142138:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4380 00000100:00080000:3.0:1787696479.142142:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696479.142145:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.142148:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.142150:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696479.142152:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696479.142159:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696479.142167:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696479.142172:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696479.142180:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696479.142187:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696479.142192:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696479.142198:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126094208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696479.142204:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126094208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696479.142224:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4390) 00000100:00100000:3.0:1787696479.653533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696479.653555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696480.677508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696480.677513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696481.061531:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299054447, at 4299054464 00000100:00100000:3.0:1787696481.701548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696481.701595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696482.725578:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696482.725625:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696483.749561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696483.749561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696484.261540:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696484.261550:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696484.261569:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696484.261580:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4385 00000100:00080000:3.0:1787696484.261591:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696484.261596:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.261601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.261605:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696484.261621:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696484.261625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.261628:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.261631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696484.261635:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696484.261648:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696484.261674:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126094336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696484.261693:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126094336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696484.261723:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696484.261727:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696484.261730:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696484.261734:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4385 00000100:00080000:3.0:1787696484.261739:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696484.261743:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.261746:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.261750:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696484.261753:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696484.261757:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.261764:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.261783:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696484.261789:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696484.261800:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696484.261808:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696484.261825:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126094464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696484.261852:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126094464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696484.261863:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696484.261867:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696484.261870:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696484.261874:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4385 00000100:00080000:3.0:1787696484.261879:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696484.261886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.261890:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.261893:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696484.261896:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696484.261901:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.261905:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.261907:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696484.261910:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696484.261918:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696484.261926:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696484.261942:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126094592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696484.261974:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126094592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696484.261994:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696484.262000:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696484.262005:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696484.262016:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4385 00000100:00080000:3.0:1787696484.262022:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696484.262030:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.262048:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.262055:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696484.262058:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696484.262065:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.262074:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.262079:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696484.262085:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696484.262091:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696484.262097:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696484.262113:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126094720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696484.262149:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126094720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696484.262168:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696484.262172:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696484.262175:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696484.262191:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4385 00000100:00080000:3.0:1787696484.262199:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696484.262204:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.262209:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.262215:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696484.262219:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696484.262227:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696484.262235:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696484.262239:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696484.262246:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696484.262253:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696484.262261:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696484.262282:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126094848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696484.262319:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126094848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696484.262336:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4395) 00000100:00100000:3.0:1787696484.773538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696484.773539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696485.797515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696485.797551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696486.181543:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299059567, at 4299059584 00000100:00100000:0.0:1787696486.821505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696486.821514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696487.845499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696487.845525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696488.869512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696488.869566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696489.381702:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.381731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126094976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696489.381765:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.381790:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696489.381838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696489.381857:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.381905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696489/real 1787696489] req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.381939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.381957:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696489.381981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696489.381988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696489.381998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382035:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126094976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696489.382050:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126094976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696489.382065:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696489.382089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696489.382116:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382125:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696489.382136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696489.382149:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382161:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696489.382167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696489/real 1787696489] req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696489.382197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696489.382201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696489.382205:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696489.382229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696489.382251:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382259:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696489.382272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696489.382280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382304:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696489.382309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696489/real 1787696489] req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382331:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696489.382337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696489.382338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696489.382341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696489.382350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696489.382365:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382370:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696489.382401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696489.382406:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382412:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696489.382440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696489/real 1787696489] req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382454:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696489.382460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696489.382461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696489.382464:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696489.382472:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382478:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696489.382486:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382491:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696489.382499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696489.382504:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696514 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382510:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696489.382513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696489/real 1787696489] req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696489.382536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696489.382541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696489.382543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696489.382545:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696489.382550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696489.382553:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126095104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696489.382558:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696489.382562:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126095232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696489.382566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696489.382569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126095360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696489.382573:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696489.382577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126095488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696514 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696489.382582:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696489.893487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696489.893522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696490.917507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696490.917514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696491.301492:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299064687, at 4299064704 00000100:00100000:3.0:1787696491.941556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696491.941597:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696492.965556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696492.965577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696493.989579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696493.989597:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696494.501748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.501779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126095616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696494.501838:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.501874:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696494.501931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696494.501968:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696494/real 1787696494] req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696494.502145:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696494.502153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696494.502167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126095616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696494.502222:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126095616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696494.502248:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696494.502269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696494.502324:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502331:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696494.502352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696494.502361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502372:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696494.502378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696494/real 1787696494] req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696494.502412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696494.502414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696494.502454:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696494.502478:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502495:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696494.502508:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502516:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696494.502526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696494.502533:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502543:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696494.502553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696494/real 1787696494] req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502574:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696494.502582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696494.502585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696494.502588:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126095872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696494.502602:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126096000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696494.502629:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502634:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696494.502641:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696494.502646:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502652:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696494.502655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696494/real 1787696494] req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502662:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696494.502672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696494.502673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696494.502675:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126096000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696494.502683:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502688:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126096128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696494.502701:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502705:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696494.502711:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696494.502715:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696519 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502720:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696494.502723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696494/real 1787696494] req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502730:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696494.502735:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696494.502740:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696494.502741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696494.502743:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696494.502747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126096128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696494.502750:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126095744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696494.502754:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696494.502759:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126095872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696494.502771:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696494.502775:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126096000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696494.502780:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696494.502782:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126096128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696519 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696494.502786:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696495.013539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696495.013540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696496.037532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696496.037573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696496.421475:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299069807, at 4299069824 00000100:00100000:0.0:1787696497.061535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696497.061554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696498.085517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696499.109516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696500.133531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696501.157588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696501.541543:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299074927, at 4299074944 00000100:00100000:0.0:1787696502.181532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696503.205534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696503.205555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696504.229535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696504.229542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696504.741557:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696504.741565:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696504.741568:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696504.741577:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4405 00000100:00080000:0.0:1787696504.741586:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696504.741588:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741591:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741594:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696504.741598:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696504.741601:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741627:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696504.741630:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741644:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696504.741664:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126096896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696504.741676:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126096896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696504.741693:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696504.741696:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696504.741697:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696504.741700:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4405 00000100:00080000:0.0:1787696504.741703:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696504.741705:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741707:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741709:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696504.741711:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696504.741714:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741716:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741717:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696504.741720:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696504.741727:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696504.741733:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696504.741744:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126097024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696504.741766:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126097024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696504.741786:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696504.741800:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696504.741803:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696504.741807:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4405 00000100:00080000:0.0:1787696504.741819:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696504.741821:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741824:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741826:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696504.741828:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696504.741829:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741832:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741836:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696504.741840:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696504.741846:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696504.741850:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696504.741861:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126097152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696504.741869:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126097152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696504.741885:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696504.741892:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696504.741895:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696504.741898:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4405 00000100:00080000:0.0:1787696504.741901:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696504.741908:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741910:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741912:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696504.741914:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696504.741916:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741918:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741920:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696504.741923:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696504.741927:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696504.741932:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696504.741943:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126097280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696504.741954:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126097280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696504.741961:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696504.741967:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696504.741969:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696504.741976:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4405 00000100:00080000:0.0:1787696504.741981:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696504.741983:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.741988:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.741993:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696504.741995:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696504.741999:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696504.742003:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696504.742006:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696504.742013:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696504.742021:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696504.742027:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696504.742036:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126097408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696504.742056:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126097408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696504.742072:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4415) 00000100:00100000:3.0:1787696505.253528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696505.253535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696506.277527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696506.277543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696506.661494:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299080047, at 4299080064 00000100:00100000:0.0:1787696507.301515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696507.301516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696508.325562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696508.325563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696509.349515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696509.349515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696510.373492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696510.373505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696511.397500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696511.397515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696511.781541:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299085167, at 4299085184 00000100:00100000:0.0:1787696512.421482:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696512.421582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696513.445533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696513.445539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696514.469499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696514.469499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696514.981577:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696514.981585:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696514.981588:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696514.981598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4415 00000100:00080000:3.0:1787696514.981607:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696514.981609:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981612:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981615:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696514.981639:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696514.981641:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981643:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981646:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696514.981648:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981658:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696514.981692:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126098176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696514.981704:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126098176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696514.981726:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696514.981729:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696514.981731:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696514.981734:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4415 00000100:00080000:3.0:1787696514.981737:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696514.981739:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981740:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981742:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696514.981744:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696514.981746:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981748:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981750:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696514.981751:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696514.981758:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696514.981761:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696514.981768:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126098304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696514.981781:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126098304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696514.981792:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696514.981797:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696514.981800:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696514.981808:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4415 00000100:00080000:3.0:1787696514.981815:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696514.981821:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981827:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981832:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696514.981837:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696514.981841:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981844:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981847:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696514.981850:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696514.981856:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696514.981861:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696514.981874:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126098432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696514.981889:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126098432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696514.981896:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696514.981898:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696514.981905:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696514.981908:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4415 00000100:00080000:3.0:1787696514.981920:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696514.981923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981926:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981929:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696514.981932:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696514.981952:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.981954:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.981955:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696514.981957:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696514.981961:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696514.981964:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696514.981974:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126098560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696514.981982:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126098560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696514.981990:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696514.981993:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696514.981994:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696514.981998:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4415 00000100:00080000:3.0:1787696514.982003:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696514.982008:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.982013:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.982018:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696514.982020:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696514.982028:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696514.982030:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696514.982032:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696514.982034:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696514.982038:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696514.982042:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696514.982052:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126098688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696514.982066:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126098688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696514.982086:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4426) 00000100:00100000:3.0:1787696515.493535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696515.493541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696516.517494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696516.517508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696516.901473:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299090287, at 4299090304 00000100:00100000:0.0:1787696517.541494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696518.565491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696518.565535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696519.589530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696519.589536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696520.101544:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696520.101555:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696520.101560:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696520.101570:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4421 00000100:00080000:0.0:1787696520.101583:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696520.101587:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.101609:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.101613:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696520.101618:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696520.101622:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.101625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.101628:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696520.101632:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696520.101645:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696520.101672:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126098816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696520.101689:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126098816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696520.101718:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696520.101728:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696520.101731:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696520.101735:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4421 00000100:00080000:0.0:1787696520.101740:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696520.101745:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.101748:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.101751:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696520.101755:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696520.101759:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.101765:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.101768:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696520.101777:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696520.101790:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696520.101808:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696520.101825:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126098944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696520.101841:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126098944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696520.101875:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696520.101881:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696520.101883:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696520.101891:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4421 00000100:00080000:0.0:1787696520.101898:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696520.101908:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.101912:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.101915:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696520.101918:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696520.101922:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.101926:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.101931:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696520.101942:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696520.101950:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696520.101958:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696520.101974:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126099072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696520.101999:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126099072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696520.102038:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696520.102051:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696520.102056:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696520.102062:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4421 00000100:00080000:0.0:1787696520.102067:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696520.102076:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.102086:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.102091:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696520.102095:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696520.102101:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.102112:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.102119:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696520.102125:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696520.102132:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696520.102142:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696520.102161:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126099200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696520.102197:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126099200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696520.102223:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696520.102233:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696520.102239:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696520.102247:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4421 00000100:00080000:0.0:1787696520.102255:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696520.102265:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.102274:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.102288:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696520.102292:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696520.102303:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696520.102313:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696520.102317:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696520.102326:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696520.102336:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696520.102345:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696520.102362:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126099328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696520.102385:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126099328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696520.102397:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4431) 00000100:00100000:3.0:1787696520.613514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696520.613529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696521.637513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696521.637527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696522.021504:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299095407, at 4299095424 00000100:00100000:3.0:1787696522.661509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696522.661514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696523.685521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696523.685521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696524.709491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696525.221524:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696525.221537:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696525.221542:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696525.221556:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4426 00000100:00080000:3.0:1787696525.221570:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696525.221573:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.221601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221606:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696525.221612:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696525.221616:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.221619:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221623:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696525.221627:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221641:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696525.221674:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126099456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696525.221695:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126099456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696525.221720:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696525.221724:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696525.221726:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696525.221731:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4426 00000100:00080000:3.0:1787696525.221735:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696525.221738:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.221742:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221745:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696525.221748:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696525.221752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.221756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221770:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696525.221774:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696525.221786:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696525.221803:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696525.221822:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126099584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696525.221835:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126099584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696525.221853:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696525.221861:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696525.221864:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696525.221869:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4426 00000100:00080000:3.0:1787696525.221875:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696525.221879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.221882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221884:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696525.221887:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696525.221891:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.221895:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221897:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696525.221901:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696525.221906:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696525.221911:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696525.221920:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126099712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696525.221932:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126099712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696525.221942:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696525.221955:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696525.221960:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696525.221970:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4426 00000100:00080000:3.0:1787696525.221979:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696525.221984:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.221988:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.221992:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696525.221997:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696525.222007:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.222015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.222020:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696525.222023:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696525.222030:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696525.222037:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696525.222056:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126099840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696525.222086:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126099840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696525.222112:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696525.222117:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696525.222120:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696525.222124:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4426 00000100:00080000:3.0:1787696525.222131:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696525.222150:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.222157:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.222171:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696525.222177:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696525.222184:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696525.222189:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696525.222195:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696525.222199:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696525.222206:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696525.222211:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696525.222222:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126099968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696525.222233:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126099968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696525.222255:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4436) 00000100:00100000:3.0:1787696525.733555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696526.757541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696527.141534:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299100527, at 4299100544 00000100:00100000:3.0:1787696527.781524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696528.805500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696529.829497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787696530.341664:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696530.341685:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696530.341688:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696530.341700:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4431 00000100:00080000:0.0:1787696530.341713:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696530.341715:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.341719:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.341723:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696530.341729:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696530.341738:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.341741:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.341767:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696530.341771:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696530.341786:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696530.341818:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126100096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696530.341834:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126100096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696530.341860:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696530.341864:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696530.341865:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696530.341868:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4431 00000100:00080000:0.0:1787696530.341872:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696530.341874:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.341876:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.341878:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696530.341880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696530.341882:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.341884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.341886:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696530.341888:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696530.341899:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696530.341903:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696530.341912:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126100224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696530.341920:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126100224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696530.341940:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696530.341943:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696530.341945:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696530.341953:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4431 00000100:00080000:0.0:1787696530.341956:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696530.341960:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.341969:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.341971:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696530.341979:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696530.341986:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.341995:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.342006:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696530.342017:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696530.342028:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696530.342038:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696530.342058:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126100352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696530.342082:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126100352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696530.342111:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696530.342121:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696530.342127:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696530.342142:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4431 00000100:00080000:0.0:1787696530.342164:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696530.342178:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.342184:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.342191:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696530.342197:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696530.342201:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.342205:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.342208:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696530.342216:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696530.342231:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696530.342239:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696530.342253:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126100480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696530.342263:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126100480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696530.342278:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696530.342282:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696530.342284:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696530.342294:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4431 00000100:00080000:0.0:1787696530.342307:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696530.342315:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.342318:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.342321:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696530.342327:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696530.342345:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696530.342359:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696530.342366:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696530.342383:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696530.342389:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696530.342404:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696530.342478:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126100608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696530.342493:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126100608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696530.342535:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4441) 00000100:00100000:3.0:1787696530.853613:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696531.877538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696532.261714:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299105647, at 4299105664 00000100:00100000:0.0:1787696532.901598:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696533.925558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696534.949563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696535.461529:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696535.461539:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696535.461544:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696535.461555:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4436 00000100:00080000:3.0:1787696535.461567:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696535.461571:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461614:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696535.461620:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696535.461623:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461626:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461629:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696535.461633:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461647:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696535.461673:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126100736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696535.461686:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126100736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696535.461704:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696535.461707:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696535.461708:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696535.461711:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4436 00000100:00080000:3.0:1787696535.461715:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696535.461717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461722:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696535.461724:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696535.461727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461731:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696535.461733:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696535.461741:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696535.461756:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696535.461766:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126100864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696535.461777:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126100864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696535.461798:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696535.461803:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696535.461805:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696535.461813:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4436 00000100:00080000:3.0:1787696535.461816:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696535.461818:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461820:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461822:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696535.461824:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696535.461826:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461828:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461829:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696535.461832:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696535.461837:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696535.461844:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696535.461854:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126100992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696535.461869:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126100992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696535.461875:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696535.461883:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696535.461885:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696535.461888:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4436 00000100:00080000:3.0:1787696535.461892:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696535.461894:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461896:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461898:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696535.461900:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696535.461907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461912:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696535.461914:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696535.461919:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696535.461923:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696535.461935:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126101120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696535.461943:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126101120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696535.461949:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696535.461952:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696535.461955:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696535.461960:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4436 00000100:00080000:3.0:1787696535.461965:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696535.461971:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461977:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.461984:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696535.461986:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696535.461991:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696535.461998:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696535.462002:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696535.462008:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696535.462017:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696535.462022:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696535.462035:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126101248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696535.462054:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126101248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696535.462089:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4446) 00000100:00100000:0.0:1787696535.973517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696536.997551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696537.381525:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299110767, at 4299110784 00000100:00100000:0.0:1787696538.021485:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696539.045580:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696540.069516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696540.581535:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696540.581547:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696540.581552:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696540.581568:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4441 00000100:00080000:3.0:1787696540.581580:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696540.581584:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.581589:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.581593:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696540.581599:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696540.581602:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.581605:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.581609:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696540.581612:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696540.581642:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696540.581668:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126101376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696540.581684:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126101376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696540.581709:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696540.581713:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696540.581715:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696540.581720:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4441 00000100:00080000:3.0:1787696540.581725:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696540.581728:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.581732:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.581735:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696540.581738:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696540.581742:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.581746:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.581749:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696540.581756:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696540.581771:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696540.581781:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696540.581796:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126101504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696540.581820:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126101504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696540.581833:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696540.581844:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696540.581846:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696540.581856:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4441 00000100:00080000:3.0:1787696540.581864:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696540.581872:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.581881:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.581885:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696540.581888:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696540.581892:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.581896:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.581902:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696540.581911:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696540.581920:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696540.581932:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696540.581950:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126101632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696540.581980:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126101632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696540.582005:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696540.582010:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696540.582014:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696540.582020:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4441 00000100:00080000:3.0:1787696540.582027:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696540.582037:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.582045:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.582058:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696540.582064:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696540.582074:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.582079:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.582082:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696540.582089:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696540.582100:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696540.582108:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696540.582136:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126101760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696540.582168:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126101760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696540.582196:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696540.582207:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696540.582212:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696540.582222:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4441 00000100:00080000:3.0:1787696540.582229:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696540.582236:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.582245:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.582252:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696540.582257:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696540.582262:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696540.582270:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696540.582277:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696540.582286:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696540.582296:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696540.582313:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696540.582331:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126101888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696540.582378:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126101888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696540.582446:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4451) 00000100:00100000:0.0:1787696541.093566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696542.117521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696542.501547:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299115887, at 4299115904 00000100:00100000:0.0:1787696543.141540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696544.165543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696545.189587:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696545.701553:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696545.701562:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696545.701565:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696545.701574:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4446 00000100:00080000:3.0:1787696545.701583:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696545.701586:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701590:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701594:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696545.701600:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696545.701603:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701606:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701610:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696545.701614:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701629:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696545.701656:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126102016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696545.701668:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126102016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696545.701689:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696545.701706:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696545.701708:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696545.701712:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4446 00000100:00080000:3.0:1787696545.701717:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696545.701719:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701722:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701724:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696545.701727:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696545.701733:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701738:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696545.701740:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696545.701748:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696545.701755:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696545.701763:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126102144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696545.701775:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126102144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696545.701790:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696545.701793:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696545.701794:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696545.701798:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4446 00000100:00080000:3.0:1787696545.701802:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696545.701805:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701809:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701829:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696545.701835:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696545.701844:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701852:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701857:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696545.701859:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696545.701864:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696545.701868:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696545.701885:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126102272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696545.701898:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126102272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696545.701912:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696545.701918:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696545.701924:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696545.701933:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4446 00000100:00080000:3.0:1787696545.701945:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696545.701947:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701952:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696545.701954:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696545.701957:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.701962:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.701966:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696545.701971:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696545.701979:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696545.701990:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696545.702010:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126102400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696545.702026:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126102400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696545.702039:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696545.702044:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696545.702048:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696545.702056:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4446 00000100:00080000:3.0:1787696545.702065:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696545.702069:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.702071:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.702073:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696545.702075:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696545.702080:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696545.702086:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696545.702090:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696545.702094:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696545.702098:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696545.702104:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696545.702119:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126102528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696545.702136:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126102528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696545.702156:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4456) 00000100:00100000:3.0:1787696546.213561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696547.237534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696547.621541:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299121007, at 4299121024 00000100:00100000:3.0:1787696548.261533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696549.285526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696550.309532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696551.333588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696552.357518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696552.741522:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299126127, at 4299126144 00000100:00100000:3.0:1787696553.381499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696553.381546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696554.405519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696554.405526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696555.429496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696555.429496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696555.941529:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696555.941554:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696555.941558:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696555.941566:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4456 00000100:00080000:3.0:1787696555.941576:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696555.941578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941582:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941584:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696555.941588:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696555.941591:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941593:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941595:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696555.941598:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941628:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696555.941654:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126103296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696555.941673:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126103296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696555.941722:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696555.941726:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696555.941728:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696555.941732:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4456 00000100:00080000:3.0:1787696555.941735:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696555.941737:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941739:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941741:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696555.941744:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696555.941746:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941748:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941750:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696555.941752:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696555.941762:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696555.941766:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696555.941775:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126103424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696555.941801:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126103424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696555.941809:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696555.941812:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696555.941813:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696555.941818:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4456 00000100:00080000:3.0:1787696555.941822:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696555.941831:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941840:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941849:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696555.941854:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696555.941864:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941871:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941890:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696555.941894:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696555.941899:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696555.941910:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696555.941918:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126103552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696555.941925:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126103552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696555.941932:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696555.941935:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696555.941937:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696555.941939:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4456 00000100:00080000:3.0:1787696555.941958:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696555.941961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941969:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696555.941971:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696555.941977:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.941979:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.941980:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696555.941993:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696555.942003:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696555.942012:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696555.942023:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126103680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696555.942031:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126103680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696555.942037:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696555.942039:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696555.942041:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696555.942046:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4456 00000100:00080000:3.0:1787696555.942048:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696555.942050:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.942052:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.942053:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696555.942054:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696555.942056:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696555.942058:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696555.942060:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696555.942061:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696555.942065:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696555.942068:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696555.942073:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126103808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696555.942085:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126103808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696555.942099:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4467) 00000100:00100000:3.0:1787696556.453551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696556.453662:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696557.477484:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696557.477490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696557.861521:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299131247, at 4299131264 00000100:00100000:0.0:1787696558.501550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696558.501550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696559.525563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696560.549527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696561.061534:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.061546:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696561.061550:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.061562:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4462 00000100:00080000:3.0:1787696561.061574:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696561.061578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061583:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061588:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696561.061594:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696561.061599:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061602:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061605:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696561.061609:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061625:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696561.061652:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126103936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696561.061669:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126103936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696561.061696:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.061700:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696561.061702:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.061718:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4462 00000100:00080000:3.0:1787696561.061723:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696561.061727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061731:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061733:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696561.061737:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696561.061740:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061744:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061747:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696561.061750:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696561.061760:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696561.061768:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696561.061787:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126104064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696561.061809:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126104064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696561.061819:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.061823:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696561.061827:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.061836:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4462 00000100:00080000:3.0:1787696561.061844:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696561.061852:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061856:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061860:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696561.061864:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696561.061868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061879:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061881:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696561.061885:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696561.061890:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696561.061895:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696561.061906:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126104192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696561.061925:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126104192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696561.061935:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.061940:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696561.061944:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.061950:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4462 00000100:00080000:3.0:1787696561.061955:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696561.061962:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061968:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061972:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696561.061975:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696561.061985:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.061988:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.061991:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696561.061997:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696561.062004:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696561.062013:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696561.062029:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126104320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696561.062054:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126104320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696561.062070:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696561.062079:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696561.062082:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696561.062088:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4462 00000100:00080000:3.0:1787696561.062095:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696561.062103:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.062111:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.062118:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696561.062121:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696561.062128:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696561.062137:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696561.062173:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696561.062180:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696561.062190:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696561.062197:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696561.062211:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126104448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696561.062225:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126104448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696561.062255:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4472) 00000100:00100000:0.0:1787696561.573537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696562.597529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696562.981505:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299136367, at 4299136384 00000100:00100000:0.0:1787696563.621569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696564.645578:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696565.669525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696566.181484:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696566.181515:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696566.181519:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696566.181536:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4467 00000100:00080000:3.0:1787696566.181546:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696566.181549:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181553:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181557:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696566.181562:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696566.181566:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181568:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181572:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696566.181576:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181590:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696566.181618:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126104576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696566.181636:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126104576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696566.181663:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696566.181667:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696566.181669:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696566.181674:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4467 00000100:00080000:3.0:1787696566.181679:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696566.181682:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181685:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181688:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696566.181692:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696566.181717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181730:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696566.181738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696566.181750:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696566.181757:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696566.181772:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126104704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696566.181801:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126104704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696566.181815:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696566.181819:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696566.181821:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696566.181827:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4467 00000100:00080000:3.0:1787696566.181833:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696566.181836:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181840:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181842:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696566.181845:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696566.181850:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181857:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181861:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696566.181865:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696566.181872:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696566.181879:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696566.181896:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126104832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696566.181916:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126104832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696566.181942:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696566.181945:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696566.181947:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696566.181953:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4467 00000100:00080000:3.0:1787696566.181957:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696566.181963:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181969:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181974:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696566.181976:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696566.181981:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.181987:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.181991:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696566.181996:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696566.182004:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696566.182009:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696566.182022:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126104960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696566.182031:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126104960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696566.182085:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696566.182089:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696566.182092:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696566.182104:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4467 00000100:00080000:3.0:1787696566.182112:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696566.182116:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.182120:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.182123:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696566.182127:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696566.182136:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696566.182143:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696566.182148:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696566.182163:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696566.182169:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696566.182177:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696566.182190:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126105088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696566.182199:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126105088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696566.182215:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4477) 00000100:00100000:0.0:1787696566.693541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696567.717509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696567.717629:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696568.101517:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299141487, at 4299141504 00000100:00100000:3.0:1787696568.741532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696568.741548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696569.765537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696569.765546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696570.789517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696570.789527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696571.301551:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696571.301567:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696571.301572:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696571.301584:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4472 00000100:00080000:3.0:1787696571.301597:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696571.301628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301632:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301635:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696571.301641:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696571.301643:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301645:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301648:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696571.301651:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301665:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696571.301695:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126105216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696571.301709:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126105216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696571.301730:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696571.301734:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696571.301735:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696571.301738:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4472 00000100:00080000:3.0:1787696571.301741:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696571.301743:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301745:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301746:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696571.301749:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696571.301751:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301752:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301754:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696571.301756:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696571.301764:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696571.301773:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696571.301781:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126105344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696571.301789:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126105344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696571.301795:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696571.301797:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696571.301799:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696571.301802:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4472 00000100:00080000:3.0:1787696571.301805:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696571.301808:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301810:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301812:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696571.301814:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696571.301818:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301821:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301823:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696571.301828:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696571.301834:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696571.301841:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696571.301850:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126105472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696571.301863:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126105472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696571.301869:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696571.301877:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696571.301879:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696571.301882:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4472 00000100:00080000:3.0:1787696571.301885:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696571.301891:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301895:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301900:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696571.301904:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696571.301910:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301915:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301919:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696571.301921:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696571.301925:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696571.301928:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696571.301936:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126105600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696571.301942:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126105600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696571.301949:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696571.301951:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696571.301952:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696571.301954:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4472 00000100:00080000:3.0:1787696571.301957:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696571.301958:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301979:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696571.301982:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696571.301987:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696571.301990:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696571.301992:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696571.301995:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696571.302004:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696571.302010:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696571.302023:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126105728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696571.302057:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126105728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696571.302068:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4482) 00000100:00100000:0.0:1787696571.813559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696571.813564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696572.837537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696573.221512:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299146607, at 4299146624 00000100:00100000:0.0:1787696573.861586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696574.885524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696574.885566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696575.909500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696575.909509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696576.421478:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696576.421487:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696576.421491:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696576.421500:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4477 00000100:00080000:3.0:1787696576.421509:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696576.421511:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421515:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421517:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696576.421521:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696576.421524:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421542:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421545:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696576.421547:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421556:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696576.421578:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126105856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696576.421590:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126105856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696576.421610:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696576.421613:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696576.421615:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696576.421618:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4477 00000100:00080000:3.0:1787696576.421620:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696576.421622:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421624:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421625:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696576.421628:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696576.421630:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421632:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421634:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696576.421636:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696576.421643:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696576.421647:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696576.421656:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126105984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696576.421685:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126105984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696576.421704:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696576.421707:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696576.421708:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696576.421713:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4477 00000100:00080000:3.0:1787696576.421716:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696576.421718:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421723:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421728:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696576.421740:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696576.421744:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421747:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421748:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696576.421750:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696576.421754:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696576.421758:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696576.421766:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126106112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696576.421776:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126106112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696576.421787:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696576.421790:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696576.421792:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696576.421800:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4477 00000100:00080000:3.0:1787696576.421806:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696576.421817:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421824:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421829:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696576.421832:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696576.421835:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421836:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421838:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696576.421840:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696576.421844:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696576.421848:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696576.421859:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126106240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696576.421895:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126106240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696576.421909:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696576.421913:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696576.421915:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696576.421923:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4477 00000100:00080000:3.0:1787696576.421932:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696576.421941:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421948:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421954:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696576.421960:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696576.421967:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696576.421974:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696576.421976:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696576.421987:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696576.421994:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696576.421999:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696576.422008:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126106368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696576.422021:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126106368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696576.422029:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4487) 00000100:00100000:0.0:1787696576.933509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696576.933512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696577.957505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696577.957505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696578.341515:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299151727, at 4299151744 00000100:00100000:0.0:1787696578.981518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696578.981522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696580.005535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696580.005540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696581.029513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696581.029520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696581.541512:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696581.541521:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696581.541524:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696581.541533:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4482 00000100:00080000:3.0:1787696581.541543:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696581.541546:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541550:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541553:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696581.541557:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696581.541560:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541562:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541565:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696581.541567:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541594:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696581.541624:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126106496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696581.541641:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126106496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696581.541667:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696581.541671:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696581.541673:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696581.541678:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4482 00000100:00080000:3.0:1787696581.541683:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696581.541687:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541692:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541694:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696581.541697:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696581.541700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541702:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541704:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696581.541709:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696581.541718:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696581.541724:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696581.541738:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126106624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696581.541747:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126106624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696581.541755:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696581.541769:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696581.541773:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696581.541781:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4482 00000100:00080000:3.0:1787696581.541799:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696581.541804:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541807:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541809:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696581.541812:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696581.541815:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541819:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541821:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696581.541828:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696581.541837:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696581.541843:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696581.541857:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126106752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696581.541867:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126106752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696581.541874:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696581.541877:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696581.541878:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696581.541884:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4482 00000100:00080000:3.0:1787696581.541890:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696581.541895:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541902:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541913:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696581.541918:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696581.541922:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.541923:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.541925:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696581.541927:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696581.541931:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696581.541938:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696581.541951:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126106880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696581.541973:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126106880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696581.541985:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696581.541990:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696581.541992:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696581.541997:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4482 00000100:00080000:3.0:1787696581.542001:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696581.542007:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.542012:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.542016:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696581.542019:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696581.542025:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696581.542032:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696581.542034:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696581.542038:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696581.542045:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696581.542060:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696581.542076:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126107008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696581.542098:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126107008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696581.542131:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4492) 00000100:00100000:0.0:1787696582.053533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696582.053533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696583.077536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696583.077536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696583.461547:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299156847, at 4299156864 00000100:00100000:0.0:1787696584.101540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696584.101540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696585.125510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696585.125550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696586.149526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696586.149530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696587.173509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696587.173509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696588.197528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696588.197634:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696588.581497:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299161967, at 4299161984 00000100:00100000:3.0:1787696589.221525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696590.245511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696591.269549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696591.781561:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696591.781575:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696591.781581:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696591.781595:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4492 00000100:00080000:3.0:1787696591.781625:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696591.781630:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781635:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781639:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696591.781646:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696591.781650:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781676:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781679:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696591.781683:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781697:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696591.781728:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126107776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696591.781745:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126107776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696591.781774:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696591.781778:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696591.781780:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696591.781784:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4492 00000100:00080000:3.0:1787696591.781788:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696591.781791:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781793:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781795:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696591.781799:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696591.781803:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781806:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781809:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696591.781812:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696591.781823:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696591.781829:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696591.781845:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126107904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696591.781863:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126107904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696591.781879:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696591.781882:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696591.781886:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696591.781897:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4492 00000100:00080000:3.0:1787696591.781905:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696591.781911:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781915:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781918:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696591.781921:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696591.781925:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781928:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781930:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696591.781933:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696591.781937:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696591.781942:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696591.781951:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126108032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696591.781958:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126108032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696591.781964:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696591.781967:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696591.781969:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696591.781972:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4492 00000100:00080000:3.0:1787696591.781975:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696591.781983:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781986:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781988:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696591.781990:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696591.781992:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.781994:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.781996:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696591.781998:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696591.782002:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696591.782009:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696591.782019:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126108160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696591.782045:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126108160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696591.782061:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696591.782064:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696591.782065:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696591.782068:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4492 00000100:00080000:3.0:1787696591.782072:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696591.782077:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.782084:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.782089:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696591.782094:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696591.782100:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696591.782102:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696591.782104:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696591.782113:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696591.782128:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696591.782136:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696591.782150:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126108288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696591.782173:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126108288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696591.782183:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4502) 00000100:00100000:3.0:1787696592.293541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696593.317569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696593.701572:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299167087, at 4299167104 00000100:00100000:3.0:1787696594.341494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696595.365513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696596.389484:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696596.901797:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.901818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126108416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696596.901850:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.901884:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696596.901924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696596.901953:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696596/real 1787696596] req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902062:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696596.902071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696596.902077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696596.902085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126108416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696596.902108:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126108416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696596.902123:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696596.902143:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696596.902180:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902210:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696596.902240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696596.902253:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902272:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696596.902282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696596/real 1787696596] req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902319:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696596.902331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696596.902336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696596.902341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696596.902377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696596.902450:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902457:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696596.902476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696596.902493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902504:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696596.902511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696596/real 1787696596] req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902537:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696596.902544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696596.902546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696596.902549:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696596.902561:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126108544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696596.902567:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696596.902573:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126108672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696596.902579:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696596.902585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902602:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696596.902612:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902619:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696596.902628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696596.902634:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902642:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696596.902646:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696596/real 1787696596] req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902663:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696596.902674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696596.902676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696596.902680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696596.902690:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696596.902710:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902717:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696596.902725:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696596.902730:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696621 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902739:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696596.902743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696596/real 1787696596] req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696596.902760:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696596.902766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696596.902768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696596.902771:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696596.902777:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126108928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696596.902781:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126108800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696596.902788:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696596.902792:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126108928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696621 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696596.902807:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696597.413508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696598.437526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696598.821529:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299172207, at 4299172224 00000100:00100000:3.0:1787696599.461585:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696600.485522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696601.509504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696602.021538:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.021545:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696602.021548:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.021557:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4503 00000100:00080000:3.0:1787696602.021572:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696602.021576:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021580:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021584:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696602.021605:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696602.021608:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021612:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696602.021614:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021624:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696602.021643:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696602.021654:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696602.021670:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.021673:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696602.021674:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696602.021677:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4503 00000100:00080000:3.0:1787696602.021681:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696602.021685:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021688:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021690:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696602.021693:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696602.021698:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696602.021700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021707:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021713:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696602.021717:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787696602.021719:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126109056:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787696602.021729:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696602.021736:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696602.021744:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696602.021751:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696602.021754:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021769:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696602.021777:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.021780:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696602.021782:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787696602.021786:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787696602.021787:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4503 00000100:00080000:3.0:1787696602.021794:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696602.021797:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021801:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696602.021806:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696602.021806:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696602.021809:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696602.021817:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021823:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021825:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696602.021827:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696602.021831:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696602.021835:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696602.021843:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787696602.021851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696602/real 1787696602] req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696602.021852:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696602.021885:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.021889:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787696602.021892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696602.021899:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787696602.021904:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696602.021909:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4503 00000100:00080000:0.0:1787696602.021913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787696602.021916:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787696602.021919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696602.021922:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696602.021926:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696602.021929:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021934:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696602.021937:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:0.0:1787696602.021941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126109056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787696602.021943:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.021948:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.021949:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:0.0:1787696602.021952:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126109056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696602.021954:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696602.021962:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696602.021967:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:0.0:1787696602.021977:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696602.021978:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696602.021987:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.021993:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696602.021995:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696602.022003:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696602.022006:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787696602.022010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109184:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787696602.022011:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4503 00000100:00080000:3.0:1787696602.022017:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696602.022019:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696602.022021:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.022022:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696602.022024:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696602.022027:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696602.022036:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696602.022040:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787696602.022044:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696602.022046:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696602.022048:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696602.022052:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787696602.022055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787696602.022057:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696602.022062:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696602.022070:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696602.022071:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696602.022079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696602/real 1787696602] req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696602.022093:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022120:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696602.022120:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4513) 00000100:00080000:0.0:1787696602.022128:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696602.022130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696602.022133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696602.022147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696602.022160:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022164:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696602.022171:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696602.022175:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022181:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696602.022184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696602/real 1787696602] req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022195:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696602.022199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696602.022201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696602.022203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696602.022210:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126109184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696602.022218:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696602.022222:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126109312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696602.022226:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696602.022239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696602.022253:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022257:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696602.022264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696602.022268:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022274:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696602.022277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696602/real 1787696602] req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696602.022292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696602.022294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696602.022296:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696602.022307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696602.022319:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022323:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696602.022329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696602.022333:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696627 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022338:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696602.022341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696602/real 1787696602] req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696602.022352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696602.022356:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696602.022358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696602.022360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696602.022363:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126109568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696602.022370:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126109440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696602.022375:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696602.022378:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126109568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696627 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696602.022382:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696602.533505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696603.557528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696603.557576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696603.941479:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299177327, at 4299177344 00000100:00100000:0.0:1787696604.581490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696604.581491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696605.605507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696605.605525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696606.629504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696606.629507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696607.141549:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696607.141558:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696607.141561:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696607.141570:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4508 00000100:00080000:0.0:1787696607.141588:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696607.141591:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141596:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141598:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696607.141603:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696607.141606:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141608:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141610:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696607.141612:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141624:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696607.141646:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126109696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696607.141659:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126109696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696607.141690:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696607.141693:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696607.141694:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696607.141698:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4508 00000100:00080000:0.0:1787696607.141701:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696607.141703:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141705:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141707:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696607.141709:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696607.141712:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141716:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141718:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696607.141720:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696607.141729:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696607.141733:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696607.141744:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126109824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696607.141755:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126109824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696607.141785:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696607.141787:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696607.141788:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696607.141791:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4508 00000100:00080000:0.0:1787696607.141793:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696607.141806:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141809:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141810:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696607.141812:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696607.141814:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141816:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141817:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696607.141819:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696607.141823:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696607.141826:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696607.141837:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126109952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696607.141864:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126109952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696607.141874:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696607.141877:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696607.141878:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696607.141880:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4508 00000100:00080000:0.0:1787696607.141894:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696607.141896:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141898:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141899:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696607.141901:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696607.141902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141904:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141908:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696607.141919:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696607.141925:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696607.141929:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696607.141941:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126110080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696607.141950:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126110080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696607.141956:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696607.141958:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696607.141959:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696607.141963:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4508 00000100:00080000:0.0:1787696607.141966:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696607.141972:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141977:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141981:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696607.141983:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696607.141989:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696607.141994:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696607.141997:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696607.142000:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696607.142004:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696607.142008:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696607.142018:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126110208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696607.142032:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126110208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696607.142055:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4518) 00000100:00100000:3.0:1787696607.653492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696607.653493:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696608.677495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696608.677499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696609.061527:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299182447, at 4299182464 00000100:00100000:0.0:1787696609.701515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696609.701539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696610.725549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696611.749503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696612.773576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696613.797506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696614.181487:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299187567, at 4299187584 00000100:00100000:0.0:1787696614.821606:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696615.845517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696616.869549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696617.381535:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696617.381546:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696617.381550:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696617.381563:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4518 00000100:00080000:3.0:1787696617.381576:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696617.381603:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381607:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381618:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696617.381624:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696617.381627:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381629:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381632:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696617.381635:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381647:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696617.381675:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126110976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696617.381689:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126110976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696617.381714:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696617.381718:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696617.381719:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696617.381722:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4518 00000100:00080000:3.0:1787696617.381726:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696617.381728:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381730:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381732:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696617.381734:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696617.381736:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381738:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381739:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696617.381741:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696617.381750:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696617.381758:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696617.381768:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126111104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696617.381780:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126111104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696617.381809:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696617.381814:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696617.381818:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696617.381824:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4518 00000100:00080000:3.0:1787696617.381828:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696617.381829:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381831:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381833:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696617.381835:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696617.381837:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381839:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381841:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696617.381843:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696617.381846:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696617.381850:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696617.381862:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126111232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696617.381869:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126111232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696617.381887:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696617.381899:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696617.381903:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696617.381907:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4518 00000100:00080000:3.0:1787696617.381911:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696617.381915:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381918:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381921:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696617.381924:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696617.381928:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.381931:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.381934:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696617.381937:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696617.381944:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696617.381949:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696617.381963:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126111360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696617.381986:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126111360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696617.382010:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696617.382014:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696617.382016:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696617.382020:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4518 00000100:00080000:3.0:1787696617.382033:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696617.382037:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.382043:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.382051:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696617.382053:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696617.382058:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696617.382063:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696617.382067:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696617.382072:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696617.382079:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696617.382082:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696617.382093:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126111488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696617.382124:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126111488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696617.382136:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4528) 00000100:00100000:0.0:1787696617.893561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696617.893622:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696618.917553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696618.917565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696619.301517:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299192687, at 4299192704 00000100:00100000:3.0:1787696619.941500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696619.941610:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696620.965523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696620.965523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696621.989519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696621.989541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696623.013524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696623.013528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696624.037490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696624.037539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696624.421533:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299197807, at 4299197824 00000100:00100000:0.0:1787696625.061510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696625.061518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696626.085497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696626.085500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696627.109517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696627.109519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696628.133483:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696628.133502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696629.157496:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696629.157498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696629.541510:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299202927, at 4299202944 00000100:00100000:0.0:1787696630.181537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696631.205557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696632.229567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696632.229594:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696633.253496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696633.253552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696634.277525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696634.277525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696634.661486:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299208047, at 4299208064 00000100:00100000:0.0:1787696635.301512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696635.301512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696636.325588:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696636.325590:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696637.349552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696637.349554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696638.373543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696638.373543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696639.397488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696639.397508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696639.397571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696639.781518:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299213167, at 4299213184 00000100:00100000:0.0:1787696640.421554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696640.421562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696641.445498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696641.445538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696642.469530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696642.469539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696642.981779:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.981817:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126114176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696642.981868:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.981908:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696642.981958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696642.981993:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696642/real 1787696642] req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982119:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696642.982193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696642.982203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696642.982221:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126114176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696642.982250:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126114176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696642.982260:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696642.982278:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696642.982301:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982307:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696642.982318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696642.982323:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982332:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696642.982337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696642/real 1787696642] req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696642.982357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696642.982359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696642.982362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696642.982382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982388:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696642.982401:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982409:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696642.982455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696642.982462:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982470:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696642.982475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696642/real 1787696642] req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982490:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696642.982496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696642.982498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696642.982500:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696642.982517:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982525:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696642.982540:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982547:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696642.982560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696642.982568:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982580:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696642.982586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696642/real 1787696642] req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982602:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696642.982607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696642.982609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696642.982611:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696642.982620:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982632:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696642.982643:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982649:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696642.982657:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696642.982662:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696667 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982669:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696642.982673:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696642/real 1787696642] req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696642.982686:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696642.982690:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696642.982692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696642.982695:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696642.982700:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126114688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696642.982703:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126114304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696642.982708:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696642.982718:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126114432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696642.982723:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696642.982744:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126114560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696642.982751:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696642.982755:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126114688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696667 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696642.982760:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696643.493491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696643.493544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696644.517562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696644.517562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696644.901577:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299218287, at 4299218304 00000100:00100000:0.0:1787696645.541500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696645.541508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696646.565565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696646.565576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696647.589514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787696648.101563:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696648.101575:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696648.101580:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696648.101591:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4549 00000100:00080000:0.0:1787696648.101604:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696648.101608:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101614:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101618:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696648.101624:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696648.101628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101631:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101635:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696648.101662:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101681:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696648.101729:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126114816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696648.101743:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126114816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696648.101764:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696648.101768:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696648.101769:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696648.101773:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4549 00000100:00080000:0.0:1787696648.101777:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696648.101780:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101783:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101785:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696648.101788:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696648.101790:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101792:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101794:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696648.101796:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696648.101806:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696648.101812:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696648.101823:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126114944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696648.101840:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126114944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696648.101847:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696648.101850:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696648.101857:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696648.101861:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4549 00000100:00080000:0.0:1787696648.101864:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696648.101870:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101878:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101883:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696648.101886:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696648.101892:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101894:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101896:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696648.101898:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696648.101903:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696648.101907:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696648.101914:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126115072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696648.101922:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126115072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696648.101928:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696648.101934:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696648.101939:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696648.101944:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4549 00000100:00080000:0.0:1787696648.101947:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696648.101951:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101954:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101958:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696648.101959:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696648.101968:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.101971:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.101973:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696648.101976:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696648.101983:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696648.101992:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696648.102006:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126115200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696648.102018:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126115200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696648.102039:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696648.102047:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696648.102051:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696648.102055:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4549 00000100:00080000:0.0:1787696648.102059:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696648.102064:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.102071:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.102076:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696648.102080:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696648.102084:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696648.102091:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696648.102096:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696648.102101:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696648.102107:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696648.102113:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696648.102129:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126115328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696648.102158:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126115328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696648.102168:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4559) 00000100:00100000:3.0:1787696648.613575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696649.637557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696650.021551:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299223407, at 4299223424 00000100:00100000:3.0:1787696650.661551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696650.661588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696651.685565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696651.685626:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696652.709503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696652.709512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696653.221758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.221784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126115456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696653.221818:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.221848:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696653.221906:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696653.221935:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.221994:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696653/real 1787696653] req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696653.222049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696653.222054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696653.222059:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126115456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696653.222084:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126115456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696653.222096:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696653.222113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696653.222140:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222148:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696653.222163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696653.222172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222183:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696653.222191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696653/real 1787696653] req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222213:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696653.222222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696653.222226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696653.222238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696653.222252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696653.222276:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222285:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696653.222298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696653.222306:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222316:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696653.222322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696653/real 1787696653] req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696653.222351:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696653.222354:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696653.222358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696653.222389:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696653.222411:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222438:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696653.222451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696653.222458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222468:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696653.222474:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696653/real 1787696653] req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222496:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696653.222504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696653.222507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696653.222511:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696653.222538:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222548:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696653.222583:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222591:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696653.222604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696653.222612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696678 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222623:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696653.222629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696653/real 1787696653] req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696653.222652:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696653.222660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696653.222664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696653.222668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696653.222676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126115968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696653.222683:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126115584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696653.222705:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696653.222712:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126115712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696653.222720:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696653.222726:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126115840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696653.222733:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696653.222738:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126115968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696678 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696653.222745:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696653.733534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696653.733538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696655.141476:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299228527, at 4299228544 00000100:00100000:0.0:1787696655.781588:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696656.805571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696657.829539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696658.341529:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696658.341538:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696658.341543:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696658.341553:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4559 00000100:00080000:0.0:1787696658.341563:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696658.341567:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.341572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341575:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696658.341581:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696658.341584:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.341587:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341591:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696658.341613:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341624:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696658.341649:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126116096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696658.341665:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126116096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696658.341690:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696658.341694:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696658.341696:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696658.341702:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4559 00000100:00080000:0.0:1787696658.341707:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696658.341710:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.341714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341717:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696658.341722:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696658.341731:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.341737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341739:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696658.341746:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696658.341762:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696658.341770:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696658.341784:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126116224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696658.341807:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126116224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696658.341833:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696658.341837:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696658.341839:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696658.341845:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4559 00000100:00080000:0.0:1787696658.341850:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696658.341853:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.341857:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341859:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696658.341862:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696658.341866:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.341870:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341873:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696658.341877:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696658.341884:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696658.341894:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696658.341911:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126116352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696658.341940:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126116352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696658.341954:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696658.341958:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696658.341960:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696658.341966:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4559 00000100:00080000:0.0:1787696658.341971:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696658.341984:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.341992:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.341998:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696658.342002:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696658.342009:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.342015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.342018:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696658.342021:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696658.342029:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696658.342040:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696658.342056:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126116480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696658.342072:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126116480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696658.342089:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696658.342094:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696658.342096:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696658.342100:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4559 00000100:00080000:0.0:1787696658.342105:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696658.342114:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.342123:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.342128:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696658.342131:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696658.342135:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696658.342143:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696658.342151:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696658.342163:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696658.342169:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696658.342180:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696658.342198:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126116608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696658.342219:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126116608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696658.342236:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4569) 00000100:00100000:0.0:1787696658.853521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696659.877601:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696660.261530:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299233647, at 4299233664 00000100:00100000:0.0:1787696660.901582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696660.901582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696661.925513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696661.925523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696662.949516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696662.949542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696663.461734:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.461755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126116736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696663.461794:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.461820:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696663.461874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696663.461899:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.461949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696663/real 1787696663] req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.461975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696663.462040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696663.462051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696663.462061:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126116736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696663.462095:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126116736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696663.462117:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696663.462142:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126116864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696663.462184:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462207:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696663.462228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696663.462238:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462251:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696663.462259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696663/real 1787696663] req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696663.462305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696663.462309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696663.462313:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462321:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126116864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696663.462327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126116992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696663.462353:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462362:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696663.462391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696663.462399:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462410:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696663.462415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696663/real 1787696663] req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462479:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696663.462487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696663.462490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696663.462494:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126116992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696663.462508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696663.462532:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462541:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696663.462555:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696663.462562:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462572:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696663.462578:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696663/real 1787696663] req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462611:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696663.462618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696663.462621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696663.462625:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696663.462638:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126116864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696663.462646:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696663.462653:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126116992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696663.462660:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696663.462665:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126117120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696663.462672:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696663.462681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696663.462712:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462719:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696663.462732:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696663.462740:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696688 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462750:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696663.462755:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696663/real 1787696663] req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696663.462778:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696663.462787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696663.462790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696663.462795:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696663.462803:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696663.462809:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126117248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696688 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696663.462817:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696663.973515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696663.973515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696664.997501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696664.997502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696665.381580:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299238767, at 4299238784 00000100:00100000:0.0:1787696666.021543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696666.021570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696667.045530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696667.045530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696668.069493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696668.069507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696668.581560:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696668.581571:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696668.581575:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696668.581587:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4569 00000100:00080000:3.0:1787696668.581598:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696668.581601:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.581606:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581610:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696668.581616:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696668.581620:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.581623:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581626:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696668.581629:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581642:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696668.581665:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696668.581677:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696668.581698:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696668.581701:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696668.581723:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696668.581728:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4569 00000100:00080000:3.0:1787696668.581733:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696668.581737:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.581741:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581744:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696668.581747:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696668.581751:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.581755:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581761:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:0.0:1787696668.581762:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696668.581767:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696668.581783:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696668.581787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126117376:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787696668.581792:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696668.581808:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696668.581816:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696668.581821:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.581837:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787696668.581846:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696668.581851:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696668.581853:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696668.581859:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4569 00000100:00080000:3.0:1787696668.581864:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696668.581871:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.581874:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581875:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000800:00000100:0.0:1787696668.581878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581878:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696668.581890:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.581895:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.581897:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:0.0:1787696668.581900:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696668.581904:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696668.581913:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696668.581920:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696668.581932:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696668.581940:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.581947:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696668/real 1787696668] req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696668.581955:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696668.581964:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696668.581967:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696668.581974:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4569 00000100:00100000:0.0:1787696668.581976:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696668.581991:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696668.581992:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696668.582003:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.582009:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.582011:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00080000:0.0:1787696668.582012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787696668.582014:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:0.0:1787696668.582019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696668.582019:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.582026:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696668.582030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696668.582031:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696668.582037:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696668.582046:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787696668.582047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126117376:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787696668.582052:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696668.582056:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126117376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696668.582065:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787696668.582080:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787696668.582091:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696668.582103:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696668.582109:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696668.582112:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787696668.582117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117504:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787696668.582117:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4569 00000100:00080000:3.0:1787696668.582127:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696668.582132:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.582136:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696668.582141:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787696668.582142:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696668.582146:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696668.582153:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696668.582159:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696668.582162:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787696668.582166:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787696668.582168:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696668.582175:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696668.582179:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:0.0:1787696668.582189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787696668.582192:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696668.582200:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696668.582205:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582216:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787696668.582218:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4579) 00000100:00100000:0.0:1787696668.582226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696668/real 1787696668] req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696668.582252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696668.582255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696668.582257:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582262:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696668.582273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696668.582307:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582312:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696668.582322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696668.582327:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582334:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696668.582337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696668/real 1787696668] req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696668.582367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696668.582370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696668.582374:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582383:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696668.582389:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126117504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696668.582404:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696668.582439:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126117632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696668.582446:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696668.582453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696668.582471:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582476:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696668.582486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696668.582491:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582498:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696668.582502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696668/real 1787696668] req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582515:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696668.582520:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696668.582522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696668.582524:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696668.582540:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696668.582554:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582559:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696668.582566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696668.582570:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696693 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582576:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696668.582588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696668/real 1787696668] req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696668.582604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696668.582609:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696668.582611:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696668.582613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696668.582618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126117888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696668.582627:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126117760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696668.582632:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696668.582636:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126117888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696693 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696668.582640:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696669.093518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696669.093521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696670.117517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696670.117520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696670.501477:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299243887, at 4299243904 00000100:00100000:3.0:1787696671.141492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696671.141501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696672.165495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696672.165495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696673.189503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696673.189507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696673.701842:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.701875:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126118016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696673.701916:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.701986:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696673.702031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696673.702055:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696673/real 1787696673] req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696673.702246:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696673.702254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696673.702268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126118016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696673.702306:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126118016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696673.702342:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696673.702369:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696673.702465:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702488:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696673.702511:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696673.702535:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702569:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696673.702581:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696673/real 1787696673] req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702613:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702697:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696673.702748:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696673.702756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696673.702768:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696673.702786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696673.702812:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702819:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696673.702832:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696673.702840:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702850:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696673.702857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696673/real 1787696673] req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702879:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696673.702894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696673.702898:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696673.702902:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702911:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696673.702918:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126118144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696673.702927:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696673.702935:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126118272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696673.702943:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696673.702954:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.702964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696673.702979:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.702988:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696673.703001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696673.703010:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.703021:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696673.703028:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696673/real 1787696673] req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.703055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.703065:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696673.703074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696673.703077:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696673.703082:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.703090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696673.703097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.703107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696673.703120:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.703129:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696673.703143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696673.703151:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696698 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.703162:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696673.703168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696673/real 1787696673] req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.703182:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696673.703212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696673.703222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696673.703226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696673.703231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696673.703239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696673.703246:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126118400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696673.703256:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696673.703263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126118528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696698 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696673.703271:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696674.213501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696674.213548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696675.237564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696675.237564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696675.621546:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299249007, at 4299249024 00000100:00100000:3.0:1787696676.261565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696676.261567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696677.285549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696677.285549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696678.309516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696678.309516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:0.0:1787696678.821748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.821770:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126118656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696678.821798:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.821828:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696678.821862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696678.821881:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.821928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696678/real 1787696678] req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.821952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.821963:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696678.821973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696678.821984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696678.821991:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126118656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696678.822034:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126118656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696678.822058:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696678.822081:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696678.822124:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822177:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696678.822192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696678.822201:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822209:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696678.822213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696678/real 1787696678] req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696678.822233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696678.822235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696678.822237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696678.822246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696678.822260:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822265:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696678.822277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696678.822282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822288:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696678.822292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696678/real 1787696678] req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822299:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696678.822310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696678.822312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696678.822314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126118912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696678.822322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126119040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696678.822336:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822340:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696678.822346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696678.822350:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822367:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696678.822370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696678/real 1787696678] req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696678.822388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696678.822389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696678.822392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126119040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696678.822399:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126118784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696678.822405:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696678.822409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126118912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696678.822414:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696678.822438:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126119040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696678.822444:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696678.822450:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126119168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696678.822470:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822475:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696678.822483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696678.822488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696703 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822494:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696678.822498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696678/real 1787696678] req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696678.822511:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696678.822515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696678.822517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696678.822519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696678.822524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126119168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696678.822528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126119168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696703 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696678.822533:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696679.333554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696679.333559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696680.357529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696680.357529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696680.741525:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299254127, at 4299254144 00000100:00100000:0.0:1787696681.381562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696681.381562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696682.405532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696682.405550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696683.429520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696683.429664:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696684.453487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696684.453487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696685.477569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696685.477569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696685.861543:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299259247, at 4299259264 00000100:00100000:0.0:1787696686.501553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696686.501553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696687.525535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696687.525558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696688.549539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696688.549539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696689.573533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696689.573533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696690.597548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696690.597562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696690.981553:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299264367, at 4299264384 00000100:00100000:3.0:1787696691.621506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696692.645530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696693.669528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696694.693548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696695.717494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696696.101531:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299269487, at 4299269504 00000100:00100000:3.0:1787696696.741530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696696.741549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696697.765518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696697.765518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696698.789547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696698.789556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696699.813552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696699.813552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696700.837565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696700.837565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696701.221502:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299274607, at 4299274624 00000100:00100000:0.0:1787696701.861492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696701.861501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696702.885525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696702.885540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696703.909479:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696704.933527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696705.957498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696706.341538:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299279727, at 4299279744 00000100:00100000:0.0:1787696706.981506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696708.005531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696709.029495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696709.029544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696709.541529:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696709.541540:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696709.541545:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696709.541556:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4610 00000100:00080000:3.0:1787696709.541569:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696709.541571:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.541575:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541588:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696709.541594:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696709.541597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.541599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541601:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696709.541604:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541620:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696709.541649:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126122496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696709.541663:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126122496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696709.541685:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696709.541708:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696709.541709:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696709.541713:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4610 00000100:00080000:3.0:1787696709.541718:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696709.541721:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.541724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541726:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696709.541729:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696709.541732:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.541735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541737:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696709.541738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696709.541749:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696709.541753:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696709.541770:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126122624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696709.541823:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126122624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696709.541855:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696709.541857:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696709.541858:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696709.541863:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4610 00000100:00080000:3.0:1787696709.541867:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696709.541870:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.541873:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541883:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696709.541889:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696709.541903:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.541915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541921:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696709.541924:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696709.541929:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696709.541933:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696709.541943:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126122752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696709.541951:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126122752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696709.541957:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696709.541960:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696709.541961:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696709.541963:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4610 00000100:00080000:3.0:1787696709.541967:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696709.541981:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.541989:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.541991:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696709.541995:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696709.541998:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.542003:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.542006:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696709.542015:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696709.542026:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696709.542036:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696709.542047:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126122880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696709.542055:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126122880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696709.542062:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696709.542064:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696709.542067:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696709.542073:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4610 00000100:00080000:3.0:1787696709.542078:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696709.542085:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.542091:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.542097:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696709.542101:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696709.542112:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696709.542118:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696709.542120:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696709.542124:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696709.542131:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696709.542137:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696709.542151:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126123008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696709.542166:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126123008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696709.542238:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4620) 00000100:00100000:0.0:1787696710.053526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696711.077569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696711.461515:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299284847, at 4299284864 00000100:00100000:3.0:1787696712.101555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696713.125501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696714.149509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696714.661572:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696714.661580:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696714.661584:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696714.661595:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4615 00000100:00080000:3.0:1787696714.661607:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696714.661610:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.661614:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.661617:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696714.661621:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696714.661623:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.661625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.661627:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696714.661629:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696714.661645:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696714.661668:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126123136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696714.661680:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126123136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696714.661699:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696714.661702:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696714.661703:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696714.661706:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4615 00000100:00080000:3.0:1787696714.661724:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696714.661731:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.661733:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.661736:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696714.661739:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696714.661741:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.661744:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.661747:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696714.661749:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696714.661757:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696714.661764:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696714.661790:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126123264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696714.661810:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126123264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696714.661822:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696714.661830:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696714.661839:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696714.661862:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4615 00000100:00080000:3.0:1787696714.661870:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696714.661876:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.661881:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.661884:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696714.661886:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696714.661888:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.661890:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.661899:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696714.661903:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696714.661911:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696714.661921:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696714.661939:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126123392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696714.661987:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126123392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696714.662020:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696714.662029:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696714.662032:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696714.662041:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4615 00000100:00080000:3.0:1787696714.662047:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696714.662050:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.662052:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.662057:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696714.662066:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696714.662077:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.662086:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.662094:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696714.662103:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696714.662108:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696714.662116:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696714.662129:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126123520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696714.662197:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126123520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696714.662274:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696714.662277:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696714.662279:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696714.662284:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4615 00000100:00080000:3.0:1787696714.662290:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696714.662293:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.662295:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.662297:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696714.662299:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696714.662301:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696714.662303:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696714.662304:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696714.662308:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696714.662316:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696714.662330:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696714.662348:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126123648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696714.662366:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126123648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696714.662400:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4625) 00000100:00100000:3.0:1787696715.173525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696716.197530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696716.581522:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299289967, at 4299289984 00000100:00100000:3.0:1787696717.221548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696718.245512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696718.245547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696719.269550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696719.269550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696720.293517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696720.293517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696721.317509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696721.317509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696721.701529:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299295087, at 4299295104 00000100:00100000:0.0:1787696722.341523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696722.341524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696723.365505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696723.365594:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696724.389512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696724.389525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696725.413481:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696725.413504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696726.437512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696726.437513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696726.821512:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299300207, at 4299300224 00000100:00100000:0.0:1787696727.461528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696727.461528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000400:00000100:3.0:1787696727.973689:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787696727.973737:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696727.973754:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696727.973801:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126124416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696749 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696727.973823:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696727.973834:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126124544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696749 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696727.973846:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696727.973868:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126124672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696749 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696727.973886:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696727.973903:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126124800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696749 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696727.973932:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696727.973967:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126124928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696749 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696728.485517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696728.485517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696729.509532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696729.509532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696730.533491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696730.533491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696731.557500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696731.557502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696731.941543:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299305327, at 4299305344 00000100:00100000:0.0:1787696732.581534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696732.581534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696733.605551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696733.605561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696734.629526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696734.629546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696735.653560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696735.653570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696736.677559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696736.677560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696737.061540:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299310447, at 4299310464 00000100:00100000:0.0:1787696737.701519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696737.701522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696738.725487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696738.725504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696739.749519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696740.261593:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696740.261608:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696740.261614:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696740.261628:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4641 00000100:00080000:0.0:1787696740.261644:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696740.261648:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.261654:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.261659:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696740.261665:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696740.261669:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.261672:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.261675:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696740.261679:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696740.261728:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696740.261763:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126126336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696740.261783:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126126336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696740.261820:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696740.261825:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696740.261827:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696740.261833:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4641 00000100:00080000:0.0:1787696740.261838:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696740.261842:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.261854:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.261857:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696740.261861:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696740.261865:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.261872:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.261879:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696740.261895:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696740.261912:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696740.261932:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696740.261953:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126126464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696740.261971:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126126464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696740.261984:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696740.261997:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696740.262001:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696740.262009:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4641 00000100:00080000:0.0:1787696740.262023:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696740.262030:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.262034:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.262037:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696740.262041:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696740.262045:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.262072:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.262081:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696740.262089:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696740.262097:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696740.262113:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696740.262133:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126126592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696740.262207:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126126592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696740.262248:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696740.262265:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696740.262271:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696740.262281:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4641 00000100:00080000:0.0:1787696740.262289:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696740.262306:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.262321:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.262346:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696740.262352:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696740.262360:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.262371:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.262384:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696740.262392:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696740.262407:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696740.262466:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696740.262496:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126126720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696740.262531:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126126720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696740.262564:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696740.262582:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696740.262587:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696740.262599:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4641 00000100:00080000:0.0:1787696740.262608:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696740.262619:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.262639:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.262646:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696740.262652:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696740.262669:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696740.262682:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696740.262692:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696740.262704:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696740.262719:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696740.262742:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696740.262763:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126126848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696740.262816:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126126848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696740.262888:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4651) 00000100:00100000:0.0:1787696740.773486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696741.797576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696742.181504:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299315567, at 4299315584 00000100:00100000:0.0:1787696742.821485:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696743.845500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696744.869474:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696745.381554:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696745.381563:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696745.381567:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696745.381576:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4646 00000100:00080000:3.0:1787696745.381586:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696745.381589:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381593:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381596:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696745.381600:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696745.381603:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381606:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381608:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696745.381611:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381636:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696745.381663:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126126976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696745.381680:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126126976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696745.381704:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696745.381708:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696745.381711:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696745.381715:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4646 00000100:00080000:3.0:1787696745.381719:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696745.381721:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381723:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381725:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696745.381727:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696745.381729:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381731:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381733:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696745.381736:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696745.381744:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696745.381748:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696745.381761:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126127104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696745.381778:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126127104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696745.381793:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696745.381813:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696745.381816:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696745.381822:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4646 00000100:00080000:3.0:1787696745.381828:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696745.381836:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381844:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381847:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696745.381851:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696745.381856:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381858:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381860:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696745.381862:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696745.381867:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696745.381871:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696745.381879:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126127232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696745.381891:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126127232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696745.381899:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696745.381901:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696745.381903:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696745.381908:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4646 00000100:00080000:3.0:1787696745.381913:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696745.381920:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381922:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381929:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696745.381931:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696745.381934:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.381936:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.381938:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696745.381941:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696745.381947:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696745.381953:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696745.381992:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126127360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696745.382025:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126127360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696745.382033:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696745.382036:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696745.382037:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696745.382044:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4646 00000100:00080000:3.0:1787696745.382050:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696745.382055:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.382058:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.382060:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696745.382064:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696745.382070:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696745.382075:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696745.382078:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696745.382083:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696745.382089:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696745.382100:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696745.382113:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126127488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696745.382143:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126127488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696745.382155:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4656) 00000100:00100000:0.0:1787696745.893540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696747.301529:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299320687, at 4299320704 00000100:00100000:3.0:1787696747.941531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696748.965576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696749.989513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696750.501567:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696750.501577:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696750.501581:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696750.501593:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4651 00000100:00080000:0.0:1787696750.501606:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696750.501608:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501612:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501616:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696750.501621:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696750.501624:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501626:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501628:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696750.501631:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501648:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696750.501676:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126127616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696750.501711:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126127616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696750.501732:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696750.501735:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696750.501736:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696750.501740:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4651 00000100:00080000:0.0:1787696750.501743:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696750.501745:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501747:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501749:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696750.501752:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696750.501755:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501757:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501759:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696750.501761:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696750.501770:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696750.501775:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696750.501786:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126127744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696750.501809:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126127744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696750.501818:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696750.501820:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696750.501823:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696750.501835:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4651 00000100:00080000:0.0:1787696750.501840:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696750.501842:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501844:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501846:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696750.501848:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696750.501851:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501852:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501854:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696750.501855:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696750.501859:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696750.501863:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696750.501889:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126127872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696750.501899:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126127872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696750.501928:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696750.501935:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696750.501937:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696750.501946:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4651 00000100:00080000:0.0:1787696750.501952:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696750.501956:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501967:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501974:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696750.501978:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696750.501981:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.501993:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.501996:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696750.501999:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696750.502005:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696750.502009:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696750.502022:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126128000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696750.502031:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126128000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696750.502037:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696750.502040:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696750.502042:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696750.502045:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4651 00000100:00080000:0.0:1787696750.502049:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696750.502054:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.502060:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.502065:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696750.502078:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696750.502093:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696750.502102:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696750.502109:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696750.502116:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696750.502125:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696750.502131:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696750.502145:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126128128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696750.502164:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126128128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696750.502182:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4661) 00000100:00100000:3.0:1787696751.013560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696752.037524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696752.421522:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299325807, at 4299325824 00000100:00100000:3.0:1787696753.061565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696754.085565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696755.109533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696755.621592:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696755.621603:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696755.621608:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696755.621618:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4656 00000100:00080000:3.0:1787696755.621629:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696755.621633:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.621638:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.621653:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696755.621658:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696755.621661:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.621665:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.621668:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696755.621671:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696755.621688:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696755.621733:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126128256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696755.621753:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126128256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696755.621814:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696755.621819:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696755.621821:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696755.621829:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4656 00000100:00080000:3.0:1787696755.621838:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696755.621852:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.621866:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.621873:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696755.621880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696755.621886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.621890:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.621893:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696755.621898:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696755.621909:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696755.621921:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696755.621948:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126128384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696755.621964:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126128384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696755.621978:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696755.621990:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696755.622000:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696755.622007:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4656 00000100:00080000:3.0:1787696755.622019:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696755.622027:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.622032:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.622036:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696755.622039:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696755.622044:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.622051:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.622061:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696755.622070:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696755.622079:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696755.622098:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696755.622125:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126128512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696755.622175:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126128512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696755.622187:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696755.622193:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696755.622195:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696755.622209:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4656 00000100:00080000:3.0:1787696755.622216:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696755.622223:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.622226:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.622229:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696755.622231:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696755.622284:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.622299:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.622304:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696755.622316:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696755.622332:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696755.622344:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696755.622371:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126128640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696755.622483:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126128640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696755.622555:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696755.622561:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696755.622564:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696755.622575:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4656 00000100:00080000:3.0:1787696755.622591:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696755.622599:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.622617:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.622632:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696755.622639:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696755.622652:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696755.622667:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696755.622680:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696755.622695:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696755.622707:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696755.622722:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696755.622743:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126128768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696755.622811:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126128768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696755.622872:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4666) 00000100:00100000:0.0:1787696756.133526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696757.157468:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696757.541484:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299330927, at 4299330944 00000100:00100000:0.0:1787696758.181497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696759.205538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696760.229503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696760.741491:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696760.741500:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696760.741504:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696760.741513:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4661 00000100:00080000:3.0:1787696760.741522:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696760.741525:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741528:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741530:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696760.741535:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696760.741537:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741539:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741541:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696760.741543:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741556:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696760.741590:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126128896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696760.741611:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126128896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696760.741637:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696760.741640:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696760.741641:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696760.741660:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4661 00000100:00080000:3.0:1787696760.741664:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696760.741667:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741669:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741671:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696760.741674:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696760.741676:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741679:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741680:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696760.741683:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696760.741691:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696760.741709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696760.741721:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126129024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696760.741736:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126129024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696760.741751:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696760.741753:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696760.741755:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696760.741758:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4661 00000100:00080000:3.0:1787696760.741761:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696760.741763:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741765:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741766:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696760.741768:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696760.741771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741788:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696760.741794:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696760.741800:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696760.741806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696760.741815:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126129152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696760.741822:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126129152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696760.741829:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696760.741831:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696760.741832:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696760.741835:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4661 00000100:00080000:3.0:1787696760.741838:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696760.741843:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741847:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741850:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696760.741852:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696760.741855:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741857:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741859:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696760.741861:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696760.741866:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696760.741871:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696760.741882:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126129280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696760.741894:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126129280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696760.741900:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696760.741902:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696760.741904:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696760.741907:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4661 00000100:00080000:3.0:1787696760.741911:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696760.741916:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741927:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696760.741931:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696760.741934:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696760.741936:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696760.741938:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696760.741940:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696760.741947:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696760.741954:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696760.741970:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126129408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696760.742008:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126129408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696760.742037:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4671) 00000100:00100000:0.0:1787696761.253517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696761.253543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696762.277496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696762.277505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696762.661538:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299336047, at 4299336064 00000100:00100000:0.0:1787696763.301525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696763.301526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696764.325508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696764.325508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696765.349525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696765.349541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:0.0:1787696765.861922:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.861952:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126129536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696765.861995:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862030:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696765.862077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696765.862104:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696765/real 1787696765] req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696765.862339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696765.862346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696765.862355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126129536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696765.862408:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126129536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696765.862475:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696765.862500:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126129664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696765.862555:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862566:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696765.862581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696765.862589:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862601:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696765.862620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696765/real 1787696765] req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696765.862662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696765.862666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696765.862670:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862679:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126129664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696765.862686:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126129792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696765.862708:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862717:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696765.862728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696765.862735:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862746:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696765.862751:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696765/real 1787696765] req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862784:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696765.862793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696765.862797:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696765.862802:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862811:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126129792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696765.862817:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126129920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696765.862839:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862848:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696765.862859:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696765.862866:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862876:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696765.862881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696765/real 1787696765] req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862905:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.862914:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696765.862922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696765.862925:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696765.862930:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126129920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696765.862944:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126129664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696765.862953:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696765.862960:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126129792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696765.862968:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696765.862972:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126129920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696765.862980:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696765.862989:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.862999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696765.863017:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.863026:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696765.863036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696765.863043:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696790 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.863053:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696765.863059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696765/real 1787696765] req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.863071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696765.863080:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696765.863089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696765.863092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696765.863096:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696765.863105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696765.863111:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126130048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696790 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696765.863119:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696766.373535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696766.373535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696767.397513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696767.397513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696767.781517:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299341167, at 4299341184 00000100:00100000:3.0:1787696768.421517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696769.445563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696770.469594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696770.981600:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696770.981615:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696770.981627:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696770.981645:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4671 00000100:00080000:3.0:1787696770.982988:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696770.983000:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983004:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983007:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696770.983011:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696770.983014:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983016:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983018:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696770.983021:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983034:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696770.983058:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696770.983070:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696770.983089:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696770.983106:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696770.983108:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696770.983113:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4671 00000100:00080000:3.0:1787696770.983118:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696770.983121:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983123:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983125:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696770.983130:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696770.983131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696770.983134:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983139:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983143:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696770.983148:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696770.983160:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787696770.983165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126130176:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787696770.983166:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696770.983180:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696770.983188:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696770.983195:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787696770.983197:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696770.983200:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696770.983203:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696770.983212:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4671 00000100:00000040:0.0:1787696770.983216:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787696770.983217:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696770.983221:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983223:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983229:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696770.983231:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696770.983233:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983235:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983237:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696770.983239:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696770.983243:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696770.983245:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696770.983252:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696770.983259:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696770.983265:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787696770.983267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787696770.983270:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696770.983273:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696770.983277:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4671 00000100:00080000:3.0:1787696770.983281:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696770.983283:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696770.983287:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696770.983289:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983294:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696770.983297:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696770.983303:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983306:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983308:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696770.983310:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696770.983314:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696770.983321:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696770.983330:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787696770.983339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696770/real 1787696770] req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696770.983341:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696770.983367:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696770.983369:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696770.983371:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696770.983376:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4671 00000100:00100000:0.0:1787696770.983377:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696770.983379:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696770.983385:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983391:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983395:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696770.983405:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696770.983406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696770.983411:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696770.983432:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696770.983435:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696770.983437:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696770.983441:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696770.983445:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696770.983451:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787696770.983470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787696770.983471:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696770.983478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787696770.983488:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4682) 00000100:00000040:0.0:1787696770.983489:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126130176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787696770.983505:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126130176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696770.983514:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787696770.983559:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696770.983594:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983600:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696770.983612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696770.983626:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983635:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787696770.983640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696770/real 1787696770] req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983654:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696770.983659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696770.983662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696770.983666:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983674:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696770.983680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983689:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696770.983711:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983719:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696770.983730:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696770.983735:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983743:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787696770.983747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696770/real 1787696770] req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983762:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983775:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696770.983784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696770.983787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696770.983792:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787696770.983808:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696770.983830:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983840:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696770.983850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696770.983855:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983863:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787696770.983869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696770/real 1787696770] req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983890:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696770.983899:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696770.983903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696770.983908:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696770.983921:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696770.983936:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983941:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787696770.983956:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787696770.983961:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696795 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.983968:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787696770.983982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787696770/real 1787696770] req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.983994:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696770.984004:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696770.984013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787696770.984017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787696770.984021:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787696770.984030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126130688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787696770.984036:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126130304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696770.984045:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696770.984052:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126130432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696770.984061:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787696770.984066:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126130560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696770.984073:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787696770.984083:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126130688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787696795 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787696770.984088:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696771.493569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696772.517552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696772.901535:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299346288, at 4299346304 00000100:00100000:3.0:1787696773.541559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696774.565500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696775.589571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696776.613523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696778.021510:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299351407, at 4299351424 00000100:00100000:3.0:1787696778.661558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696779.685506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696780.709526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696781.221559:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696781.221567:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696781.221571:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696781.221582:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4682 00000100:00080000:0.0:1787696781.221594:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696781.221598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.221602:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221607:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696781.221612:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696781.221615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.221618:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221622:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696781.221625:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221640:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696781.221663:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126131456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696781.221675:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126131456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696781.221694:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696781.221697:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696781.221699:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696781.221715:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4682 00000100:00080000:0.0:1787696781.221724:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696781.221727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.221730:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221732:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696781.221735:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696781.221737:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.221739:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221741:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696781.221744:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696781.221752:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696781.221757:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696781.221773:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126131584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696781.221796:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126131584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696781.221808:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696781.221814:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696781.221818:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696781.221829:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4682 00000100:00080000:0.0:1787696781.221837:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696781.221842:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.221845:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221848:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696781.221851:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696781.221854:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.221871:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221874:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696781.221879:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696781.221887:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696781.221894:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696781.221908:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126131712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696781.221924:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126131712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696781.221938:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696781.221946:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696781.221951:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696781.221961:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4682 00000100:00080000:0.0:1787696781.221968:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696781.221977:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.221985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.221988:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696781.221991:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696781.221998:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.222004:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.222011:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696781.222018:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696781.222027:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696781.222041:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696781.222059:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126131840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696781.222088:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126131840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696781.222101:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696781.222109:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696781.222112:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696781.222123:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4682 00000100:00080000:0.0:1787696781.222132:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696781.222137:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.222145:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.222150:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696781.222153:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696781.222160:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696781.222169:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696781.222174:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696781.222178:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696781.222193:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696781.222203:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696781.222219:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126131968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696781.222253:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126131968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696781.222271:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4692) 00000100:00100000:3.0:1787696781.733502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696782.757532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696782.757568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696783.141500:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299356527, at 4299356544 00000100:00100000:0.0:1787696783.781499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696783.781502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696784.805524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696784.805530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696785.829517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696785.829518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696786.341532:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696786.341543:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696786.341547:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696786.341559:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4687 00000100:00080000:3.0:1787696786.341571:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696786.341574:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.341579:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.341584:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696786.341589:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696786.341593:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.341596:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.341599:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696786.341604:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696786.341620:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696786.341662:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126132096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696786.341679:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126132096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696786.341704:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696786.341709:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696786.341711:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696786.341716:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4687 00000100:00080000:3.0:1787696786.341721:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696786.341724:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.341728:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.341732:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696786.341735:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696786.341739:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.341743:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.341746:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696786.341749:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696786.341761:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696786.341771:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696786.341788:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126132224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696786.341806:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126132224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696786.341832:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696786.341837:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696786.341845:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696786.341851:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4687 00000100:00080000:3.0:1787696786.341858:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696786.341864:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.341869:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.341874:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696786.341879:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696786.341886:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.341890:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.341893:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696786.341896:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696786.341903:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696786.341911:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696786.341930:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126132352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696786.341947:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126132352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696786.341965:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696786.341971:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696786.341976:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696786.341986:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4687 00000100:00080000:3.0:1787696786.341994:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696786.342003:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.342012:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.342018:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696786.342021:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696786.342039:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.342048:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.342052:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696786.342056:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696786.342065:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696786.342074:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696786.342091:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126132480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696786.342112:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126132480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696786.342130:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696786.342138:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696786.342142:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696786.342149:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4687 00000100:00080000:3.0:1787696786.342155:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696786.342163:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.342171:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.342177:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696786.342183:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696786.342194:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696786.342202:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696786.342210:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696786.342219:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696786.342230:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696786.342242:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696786.342289:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126132608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696786.342329:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126132608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696786.342366:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4697) 00000100:00100000:3.0:1787696786.853506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696786.853507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696787.877524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696787.877525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696788.261531:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299361647, at 4299361664 00000100:00100000:0.0:1787696788.901518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696789.925545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696789.925595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696790.949479:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696790.949496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696791.973513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696791.973513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696792.997497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696792.997497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696793.381523:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299366767, at 4299366784 00000100:00100000:0.0:1787696794.021545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696794.021552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000400:00000100:0.0:1787696794.533643:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787696794.533684:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787696794.533698:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787696794.533730:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126132736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696816 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787696794.533746:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787696794.533752:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126132864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696816 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787696794.533758:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787696794.533763:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126132992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696816 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787696794.533769:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787696794.533774:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126133120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696816 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787696794.533781:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787696794.533788:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126133248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696816 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696795.045527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696795.045528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696795.045587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696796.069536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696796.069536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696797.093491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696797.093492:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696798.117516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696798.117526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696798.501515:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299371887, at 4299371904 00000100:00100000:0.0:1787696799.141499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696799.141499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696800.165525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696800.165532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696801.189486:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696801.189486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696802.213558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696802.213565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696803.237476:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696803.237518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696803.621536:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299377007, at 4299377024 00000100:00100000:0.0:1787696804.261521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696805.285558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696806.309630:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787696806.821510:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696806.821517:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696806.821521:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696806.821529:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4707 00000100:00080000:0.0:1787696806.821538:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696806.821541:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821544:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821548:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696806.821553:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696806.821557:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821562:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696806.821566:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821579:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696806.821604:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126134656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696806.821618:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126134656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696806.821638:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696806.821653:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696806.821661:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696806.821666:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4707 00000100:00080000:0.0:1787696806.821673:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696806.821676:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821679:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821682:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696806.821685:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696806.821688:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821690:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821692:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696806.821694:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696806.821704:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696806.821709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696806.821719:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126134784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696806.821730:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126134784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696806.821748:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696806.821752:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696806.821756:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696806.821765:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4707 00000100:00080000:0.0:1787696806.821768:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696806.821771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821782:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696806.821785:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696806.821791:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821796:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821798:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696806.821801:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696806.821805:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696806.821810:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696806.821818:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126134912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696806.821827:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126134912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696806.821834:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696806.821837:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696806.821839:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696806.821843:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4707 00000100:00080000:0.0:1787696806.821846:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696806.821849:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821852:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821853:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696806.821854:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696806.821856:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821858:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821859:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696806.821860:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696806.821864:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696806.821870:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696806.821876:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126135040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696806.821882:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126135040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696806.821887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696806.821889:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696806.821890:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696806.821896:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4707 00000100:00080000:0.0:1787696806.821912:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696806.821914:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821916:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821918:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696806.821919:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696806.821921:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696806.821922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696806.821925:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696806.821929:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696806.821934:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696806.821938:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696806.821945:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126135168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696806.821951:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126135168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696806.821957:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4717) 00000100:00100000:0.0:1787696807.333522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696808.357482:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696808.741493:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299382127, at 4299382144 00000100:00100000:0.0:1787696809.381523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696810.405508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696811.429651:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696811.429796:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696812.453549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696812.453549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696813.477554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696813.477554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696813.861513:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299387247, at 4299387264 00000100:00100000:0.0:1787696814.501521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696814.501521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696815.525526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696815.525526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696816.549516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696816.549616:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696817.061599:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696817.061611:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696817.061616:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696817.061628:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4718 00000100:00080000:0.0:1787696817.061641:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696817.061645:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.061650:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.061655:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696817.061661:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696817.061665:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.061668:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.061671:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696817.061675:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696817.061692:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696817.061734:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126135936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696817.061752:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126135936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696817.061774:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696817.061794:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696817.061796:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696817.061801:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4718 00000100:00080000:0.0:1787696817.061806:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696817.061809:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.061813:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.061815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696817.061819:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696817.061822:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.061825:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.061828:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696817.061831:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696817.061842:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696817.061846:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696817.061858:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126136064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696817.061875:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126136064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696817.061912:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696817.061916:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696817.061920:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696817.061933:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4718 00000100:00080000:0.0:1787696817.061937:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696817.061940:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.061942:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.061950:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696817.061953:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696817.061957:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.061960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.061963:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696817.061970:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696817.061980:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696817.061993:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696817.062012:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126136192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696817.062024:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126136192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696817.062034:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696817.062038:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696817.062040:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696817.062051:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4718 00000100:00080000:0.0:1787696817.062057:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696817.062060:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.062064:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.062067:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696817.062072:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696817.062084:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.062096:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.062105:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696817.062109:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696817.062114:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696817.062123:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696817.062137:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126136320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696817.062185:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126136320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696817.062196:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696817.062202:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696817.062206:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696817.062219:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4718 00000100:00080000:0.0:1787696817.062228:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696817.062236:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.062245:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.062250:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696817.062256:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696817.062263:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696817.062270:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696817.062275:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696817.062283:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696817.062292:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696817.062304:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696817.062322:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126136448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696817.062333:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126136448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696817.062353:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4728) 00000100:00100000:0.0:1787696817.573570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696817.573571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696818.597531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696818.597531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696818.981568:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299392367, at 4299392384 00000100:00100000:3.0:1787696819.621489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696819.621533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696820.645515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696820.645551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696821.669695:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696821.669787:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696821.671630:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696822.693472:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696822.693487:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696823.717502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696823.717502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696824.101507:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299397487, at 4299397504 00000100:00100000:0.0:1787696824.741519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696824.741519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696825.765530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696826.789605:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696827.813513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696828.837576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696829.221563:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299402607, at 4299402624 00000100:00100000:3.0:1787696829.861560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696830.885556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696831.909528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696832.421575:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696832.421586:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696832.421590:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696832.421600:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4733 00000100:00080000:0.0:1787696832.421611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696832.421615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.421620:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.421624:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696832.421629:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696832.421632:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.421635:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.421639:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696832.421642:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696832.421657:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696832.421682:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126137856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696832.421699:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126137856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696832.421729:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696832.421733:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696832.421749:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696832.421754:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4733 00000100:00080000:0.0:1787696832.421759:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696832.421762:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.421766:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.421769:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696832.421773:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696832.421776:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.421782:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.421787:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696832.421794:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696832.421807:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696832.421815:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696832.421829:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126137984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696832.421856:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126137984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696832.421868:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696832.421872:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696832.421874:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696832.421880:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4733 00000100:00080000:0.0:1787696832.421888:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696832.421892:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.421896:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.421901:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696832.421904:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696832.421918:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.421923:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.421925:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696832.421929:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696832.421936:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696832.421943:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696832.421959:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126138112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696832.421996:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126138112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696832.422028:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696832.422035:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696832.422040:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696832.422049:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4733 00000100:00080000:0.0:1787696832.422055:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696832.422062:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.422069:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.422075:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696832.422079:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696832.422085:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.422090:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.422092:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696832.422095:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696832.422102:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696832.422112:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696832.422124:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126138240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696832.422159:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126138240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696832.422177:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696832.422196:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696832.422200:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696832.422206:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4733 00000100:00080000:0.0:1787696832.422212:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696832.422219:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.422227:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.422230:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696832.422234:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696832.422242:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696832.422250:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696832.422256:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696832.422262:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696832.422271:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696832.422280:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696832.422294:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126138368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696832.422319:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126138368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696832.422341:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4743) 00000100:00100000:3.0:1787696832.933547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696832.933621:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696833.957531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696833.957577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696834.341555:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299407727, at 4299407744 00000100:00100000:3.0:1787696834.981543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696834.981554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696836.005526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696836.005530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696837.029534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696837.029540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696838.053519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696838.053519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696839.077529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696839.077529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696839.461501:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299412847, at 4299412864 00000100:00100000:3.0:1787696840.101513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696841.125518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696842.149526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696842.661562:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696842.661570:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696842.661573:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696842.661583:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4743 00000100:00080000:0.0:1787696842.661594:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696842.661596:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661603:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696842.661607:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696842.661610:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661612:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661614:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696842.661617:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661630:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696842.661654:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126139136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696842.661665:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126139136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696842.661686:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696842.661689:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696842.661690:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696842.661693:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4743 00000100:00080000:0.0:1787696842.661696:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696842.661712:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661716:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696842.661719:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696842.661722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661726:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696842.661727:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696842.661735:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696842.661740:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696842.661749:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126139264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696842.661763:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126139264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696842.661775:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696842.661779:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696842.661780:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696842.661783:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4743 00000100:00080000:0.0:1787696842.661786:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696842.661788:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661790:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661792:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696842.661793:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696842.661795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661799:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696842.661809:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696842.661816:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696842.661821:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696842.661832:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126139392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696842.661840:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126139392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696842.661846:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696842.661848:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696842.661850:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696842.661853:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4743 00000100:00080000:0.0:1787696842.661856:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696842.661857:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661861:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696842.661864:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696842.661868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661872:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661874:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696842.661876:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696842.661880:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696842.661883:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696842.661891:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126139520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696842.661914:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126139520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696842.661925:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696842.661927:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696842.661929:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696842.661932:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4743 00000100:00080000:0.0:1787696842.661935:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696842.661937:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661939:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661941:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696842.661943:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696842.661948:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696842.661954:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696842.661958:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696842.661964:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696842.661971:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696842.661974:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696842.661982:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126139648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696842.662005:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126139648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696842.662013:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4753) 00000100:00100000:3.0:1787696843.173529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696844.197522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696844.581473:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299417967, at 4299417984 00000100:00100000:3.0:1787696845.221524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696846.245530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696847.269479:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696847.269512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787696847.781519:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696847.781527:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696847.781530:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696847.781539:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4748 00000100:00080000:0.0:1787696847.781548:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696847.781565:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781569:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781571:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696847.781576:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696847.781578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781580:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781583:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696847.781587:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781595:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696847.781618:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126139776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696847.781630:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126139776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696847.781650:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696847.781653:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696847.781655:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696847.781658:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4748 00000100:00080000:0.0:1787696847.781661:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696847.781662:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781664:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781665:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696847.781667:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696847.781670:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781671:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781673:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696847.781675:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696847.781683:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696847.781691:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696847.781700:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126139904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696847.781707:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126139904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696847.781715:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696847.781718:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696847.781720:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696847.781736:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4748 00000100:00080000:0.0:1787696847.781742:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696847.781749:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781752:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781753:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696847.781755:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696847.781757:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781759:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781760:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696847.781762:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696847.781766:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696847.781769:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696847.781780:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126140032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696847.781788:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126140032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696847.781799:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696847.781808:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696847.781811:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696847.781817:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4748 00000100:00080000:0.0:1787696847.781821:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696847.781823:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781827:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696847.781829:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696847.781831:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781832:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781834:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696847.781835:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696847.781839:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696847.781844:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696847.781850:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126140160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696847.781862:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126140160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696847.781869:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696847.781872:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696847.781873:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696847.781876:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4748 00000100:00080000:0.0:1787696847.781879:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696847.781881:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781885:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781893:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696847.781897:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696847.781904:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696847.781910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696847.781912:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696847.781914:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696847.781917:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696847.781921:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696847.781928:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126140288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696847.781935:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126140288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696847.781945:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4758) 00000100:00100000:3.0:1787696848.293529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696848.293549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696849.317520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696849.317566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696849.701531:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299423087, at 4299423104 00000100:00100000:3.0:1787696850.341576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696850.341598:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696851.365508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696851.365508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696852.389526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696852.389526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696853.413537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696853.413572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696854.437554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696854.821503:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299428207, at 4299428224 00000100:00100000:3.0:1787696855.461524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696856.485540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696857.509516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696858.533572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696859.557512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696859.941554:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299433327, at 4299433344 00000100:00100000:3.0:1787696860.581568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000400:00000100:3.0:1787696861.093697:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787696861.093736:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696861.093753:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696861.093812:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126141056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787696883 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696861.093833:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696861.093844:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126141184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696883 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696861.093856:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696861.093867:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126141312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787696883 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696861.093879:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696861.093890:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126141440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696883 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696861.093904:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696861.093915:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126141568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787696883 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696861.605532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696861.605582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696862.629536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696862.629536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696863.653524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696863.653525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696864.677581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696864.677591:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696865.061865:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299438447, at 4299438464 00000100:00100000:3.0:1787696865.701527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696865.701528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696866.725493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696866.725531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696867.749546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696867.749548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696868.773545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696868.773548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696869.797577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696869.797600:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696870.181511:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299443567, at 4299443584 00000100:00100000:3.0:1787696870.821489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696870.821533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696871.845586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696871.845594:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696872.869538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696872.869545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696873.893532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696873.893554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696874.917520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696874.917530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696875.301562:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299448687, at 4299448704 00000100:00100000:3.0:1787696875.941580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696876.965519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696877.989558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696879.013566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696880.037557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696880.421520:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299453807, at 4299453824 00000100:00100000:3.0:1787696881.061549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696882.085633:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696883.109552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696883.621549:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696883.621561:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696883.621566:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696883.621587:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4784 00000100:00080000:3.0:1787696883.621600:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696883.621604:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.621609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621612:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696883.621618:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696883.621622:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.621624:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621627:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696883.621630:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621648:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696883.621681:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126144256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696883.621718:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126144256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696883.621745:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696883.621749:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696883.621750:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696883.621760:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4784 00000100:00080000:3.0:1787696883.621764:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696883.621767:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.621771:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621773:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696883.621776:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696883.621780:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.621783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621785:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696883.621788:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696883.621799:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696883.621806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696883.621826:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126144384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696883.621838:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126144384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696883.621847:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696883.621850:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696883.621852:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696883.621860:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4784 00000100:00080000:3.0:1787696883.621871:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696883.621878:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.621885:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621890:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696883.621893:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696883.621896:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.621899:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621901:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696883.621904:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696883.621909:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696883.621914:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696883.621924:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126144512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696883.621945:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126144512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696883.621955:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696883.621959:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696883.621960:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696883.621965:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4784 00000100:00080000:3.0:1787696883.621971:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696883.621978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.621984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.621987:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696883.621990:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696883.621993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.622000:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.622003:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696883.622005:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696883.622011:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696883.622015:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696883.622027:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126144640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696883.622049:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126144640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696883.622070:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696883.622074:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696883.622078:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696883.622086:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4784 00000100:00080000:3.0:1787696883.622092:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696883.622099:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.622108:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.622114:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696883.622117:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696883.622122:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696883.622126:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696883.622130:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696883.622137:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696883.622145:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696883.622151:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696883.622161:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126144768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696883.622190:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126144768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696883.622205:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4794) 00000100:00100000:3.0:1787696884.133514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696885.157558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696885.541566:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299458927, at 4299458944 00000100:00100000:3.0:1787696886.181640:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696887.205497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696887.205497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696888.229483:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696888.229483:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696888.741570:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696888.741580:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696888.741585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696888.741598:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4789 00000100:00080000:3.0:1787696888.741613:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696888.741617:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741621:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741626:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696888.741632:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696888.741635:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741637:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741639:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696888.741641:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741662:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696888.741691:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126144896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696888.741704:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126144896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696888.741744:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696888.741748:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696888.741750:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696888.741755:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4789 00000100:00080000:3.0:1787696888.741759:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696888.741762:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741769:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696888.741772:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696888.741775:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741779:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696888.741784:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696888.741793:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696888.741798:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696888.741810:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126145024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696888.741817:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126145024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696888.741823:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696888.741825:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696888.741827:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696888.741829:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4789 00000100:00080000:3.0:1787696888.741832:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696888.741843:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741852:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696888.741860:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696888.741862:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741865:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696888.741866:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696888.741870:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696888.741874:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696888.741881:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126145152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696888.741887:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126145152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696888.741893:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696888.741895:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696888.741896:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696888.741902:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4789 00000100:00080000:3.0:1787696888.741906:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696888.741913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741916:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696888.741918:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696888.741920:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.741925:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.741930:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696888.741937:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696888.741942:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696888.741952:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696888.741960:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126145280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696888.741966:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126145280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696888.741971:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696888.741975:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696888.741978:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696888.741983:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4789 00000100:00080000:3.0:1787696888.741988:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696888.741996:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.742002:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.742006:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696888.742010:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696888.742015:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696888.742020:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696888.742023:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696888.742028:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696888.742033:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696888.742037:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696888.742043:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126145408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696888.742056:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126145408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696888.742076:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4799) 00000100:00100000:0.0:1787696889.253558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696889.253558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696890.277548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696890.663690:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299464047, at 4299464066 00000100:00100000:0.0:1787696891.301724:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696892.325533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696893.349560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696893.861563:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696893.861575:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696893.861580:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696893.861593:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4794 00000100:00080000:3.0:1787696893.861605:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696893.861609:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.861613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.861617:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696893.861623:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696893.861626:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.861629:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.861632:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696893.861635:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696893.861653:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696893.861684:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126145536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696893.861701:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126145536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696893.861727:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696893.861730:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696893.861732:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696893.861736:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4794 00000100:00080000:3.0:1787696893.861758:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696893.861761:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.861766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.861769:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696893.861772:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696893.861775:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.861779:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.861781:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696893.861784:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696893.861794:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696893.861800:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696893.861818:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126145664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696893.861849:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126145664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696893.861877:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696893.861880:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696893.861882:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696893.861886:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4794 00000100:00080000:3.0:1787696893.861889:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696893.861892:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.861895:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.861897:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696893.861899:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696893.861902:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.861907:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.861920:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696893.861928:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696893.861942:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696893.861949:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696893.861963:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126145792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696893.861974:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126145792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696893.861987:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696893.861997:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696893.862004:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696893.862011:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4794 00000100:00080000:3.0:1787696893.862016:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696893.862021:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.862026:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.862028:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696893.862033:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696893.862038:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.862042:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.862045:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696893.862051:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696893.862060:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696893.862068:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696893.862084:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126145920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696893.862158:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126145920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696893.862177:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696893.862190:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696893.862193:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696893.862211:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4794 00000100:00080000:3.0:1787696893.862220:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696893.862227:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.862234:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.862237:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696893.862240:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696893.862243:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696893.862245:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696893.862247:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696893.862250:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696893.862259:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696893.862268:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696893.862284:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126146048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696893.862304:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126146048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696893.862352:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4804) 00000100:00100000:0.0:1787696894.373519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696895.397543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696895.781502:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299469167, at 4299469184 00000100:00100000:0.0:1787696896.421502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696897.445512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696897.445536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696898.469499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696898.469508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696898.981474:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696898.981482:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696898.981486:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696898.981497:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4799 00000100:00080000:3.0:1787696898.981508:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696898.981513:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981518:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981522:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696898.981528:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696898.981532:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981535:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981538:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696898.981541:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981557:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696898.981582:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126146176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696898.981596:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126146176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696898.981621:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696898.981625:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696898.981627:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696898.981631:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4799 00000100:00080000:3.0:1787696898.981634:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696898.981638:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981641:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981658:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696898.981664:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696898.981673:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981681:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981684:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696898.981691:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696898.981702:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696898.981708:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696898.981724:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126146304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696898.981745:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126146304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696898.981755:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696898.981759:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696898.981761:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696898.981767:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4799 00000100:00080000:3.0:1787696898.981773:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696898.981779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981786:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696898.981789:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696898.981794:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981801:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981807:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696898.981814:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696898.981823:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696898.981836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696898.981852:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126146432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696898.981868:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126146432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696898.981885:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696898.981891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696898.981895:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696898.981905:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4799 00000100:00080000:3.0:1787696898.981915:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696898.981925:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981934:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981951:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696898.981956:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696898.981961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.981967:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.981972:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696898.981978:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696898.981988:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696898.981996:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696898.982010:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126146560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696898.982035:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126146560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696898.982043:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696898.982050:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696898.982052:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696898.982058:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4799 00000100:00080000:3.0:1787696898.982065:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696898.982070:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.982075:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.982077:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696898.982079:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696898.982084:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696898.982089:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696898.982093:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696898.982098:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696898.982104:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696898.982109:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696898.982118:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126146688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696898.982127:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126146688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696898.982134:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4810) 00000100:00100000:0.0:1787696899.493580:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696899.493580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696900.517492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696900.517518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696900.901537:0:70704:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299474287, at 4299474304 00000100:00100000:0.0:1787696901.541547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696901.541557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696902.565527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696902.565627:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696903.589537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696903.589537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696904.101521:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696904.101530:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696904.101551:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696904.101560:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4805 00000100:00080000:3.0:1787696904.101570:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696904.101573:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101576:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101579:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696904.101583:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696904.101586:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101587:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101597:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696904.101600:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101611:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696904.101635:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126146816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696904.101647:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126146816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696904.101666:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696904.101669:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696904.101670:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696904.101674:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4805 00000100:00080000:3.0:1787696904.101676:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696904.101678:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101679:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101681:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696904.101683:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696904.101685:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101687:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101696:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696904.101699:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696904.101708:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696904.101711:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696904.101723:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126146944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696904.101736:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126146944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696904.101747:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696904.101749:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696904.101751:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696904.101754:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4805 00000100:00080000:3.0:1787696904.101758:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696904.101759:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101761:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101763:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696904.101765:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696904.101771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101776:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101781:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696904.101785:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696904.101792:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696904.101798:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696904.101806:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126147072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696904.101821:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126147072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696904.101828:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696904.101833:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696904.101837:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696904.101840:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4805 00000100:00080000:3.0:1787696904.101845:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696904.101847:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101853:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101858:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696904.101862:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696904.101868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101871:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101873:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696904.101875:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696904.101879:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696904.101883:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696904.101889:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126147200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696904.101901:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126147200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696904.101935:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696904.101938:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696904.101939:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696904.101952:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4805 00000100:00080000:3.0:1787696904.101957:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696904.101963:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101971:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101975:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696904.101978:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696904.101982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696904.101988:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696904.101990:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696904.101992:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696904.101996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696904.102001:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696904.102010:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126147328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696904.102024:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126147328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696904.102045:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4815) 00000100:00100000:0.0:1787696904.613525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696905.637534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696906.021500:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299479407, at 4299479424 00000100:00100000:0.0:1787696906.661527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696907.685522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696908.709550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696909.221491:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696909.221499:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696909.221502:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696909.221511:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4810 00000100:00080000:3.0:1787696909.221521:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696909.221524:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221527:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221530:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696909.221549:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696909.221552:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221554:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221557:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696909.221559:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221568:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696909.221603:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126147456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696909.221619:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126147456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696909.221643:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696909.221647:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696909.221649:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696909.221653:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4810 00000100:00080000:3.0:1787696909.221657:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696909.221661:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221664:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221667:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696909.221670:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696909.221674:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221677:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221680:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696909.221683:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696909.221695:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696909.221702:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696909.221719:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126147584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696909.221749:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126147584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696909.221759:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696909.221762:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696909.221763:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696909.221767:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4810 00000100:00080000:3.0:1787696909.221770:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696909.221772:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221774:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221776:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696909.221778:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696909.221781:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221782:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221784:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696909.221786:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696909.221790:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696909.221795:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696909.221805:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126147712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696909.221812:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126147712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696909.221821:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696909.221829:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696909.221835:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696909.221839:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4810 00000100:00080000:3.0:1787696909.221843:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696909.221845:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221848:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221849:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696909.221851:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696909.221857:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221864:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221868:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696909.221874:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696909.221879:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696909.221883:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696909.221892:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126147840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696909.221918:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126147840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696909.221931:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696909.221936:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696909.221939:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696909.221953:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4810 00000100:00080000:3.0:1787696909.221958:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696909.221965:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221972:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221974:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696909.221977:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696909.221987:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696909.221989:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696909.221991:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696909.221992:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696909.221996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696909.221999:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696909.222013:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126147968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696909.222023:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126147968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696909.222046:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4820) 00000100:00100000:0.0:1787696909.733496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696910.757532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696911.141540:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299484527, at 4299484544 00000100:00100000:0.0:1787696911.781523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696911.781544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696912.805531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696912.805531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696913.829521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696914.341549:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696914.341561:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696914.341565:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696914.341598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4815 00000100:00080000:0.0:1787696914.341610:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696914.341614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.341618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.341622:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696914.341627:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696914.341631:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.341633:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.341637:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696914.341640:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696914.341653:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696914.341681:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126148096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696914.341697:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126148096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696914.341722:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696914.341727:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696914.341729:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696914.341733:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4815 00000100:00080000:0.0:1787696914.341738:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696914.341741:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.341745:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.341748:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696914.341751:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696914.341755:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.341759:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.341768:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696914.341776:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696914.341787:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696914.341794:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696914.341810:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126148224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696914.341823:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126148224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696914.341838:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696914.341847:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696914.341851:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696914.341860:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4815 00000100:00080000:0.0:1787696914.341865:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696914.341869:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.341872:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.341875:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696914.341878:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696914.341882:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.341885:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.341887:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696914.341890:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696914.341896:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696914.341903:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696914.341915:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126148352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696914.341956:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126148352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696914.341967:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696914.341971:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696914.341976:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696914.341985:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4815 00000100:00080000:0.0:1787696914.341994:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696914.342001:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.342009:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.342015:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696914.342019:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696914.342022:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.342025:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.342031:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696914.342035:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696914.342064:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696914.342071:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696914.342086:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126148480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696914.342110:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126148480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696914.342122:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696914.342131:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696914.342134:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696914.342156:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4815 00000100:00080000:0.0:1787696914.342164:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696914.342171:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.342178:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.342183:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696914.342186:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696914.342189:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696914.342193:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696914.342198:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696914.342205:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696914.342216:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696914.342224:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696914.342243:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126148608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696914.342271:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126148608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696914.342288:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4825) 00000100:00100000:3.0:1787696914.853548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696915.877524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696915.877550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696916.261580:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299489647, at 4299489664 00000100:00100000:0.0:1787696916.901540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696917.925508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696918.949568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696919.973535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696920.997519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696921.381523:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299494767, at 4299494784 00000100:00100000:3.0:1787696922.021571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696923.045522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696924.069526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696925.093478:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696926.117534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696926.501524:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299499887, at 4299499904 00000100:00100000:3.0:1787696927.141500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696928.165534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696929.189533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696929.701583:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696929.701591:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696929.701595:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696929.701603:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4830 00000100:00080000:0.0:1787696929.701612:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696929.701615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701618:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701633:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696929.701638:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696929.701640:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701642:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701644:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696929.701647:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701655:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696929.701679:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126150016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696929.701695:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126150016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696929.701720:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696929.701723:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696929.701725:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696929.701728:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4830 00000100:00080000:0.0:1787696929.701732:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696929.701734:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701736:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701738:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696929.701740:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696929.701742:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701744:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701746:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696929.701749:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696929.701757:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696929.701761:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696929.701792:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126150144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696929.701805:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126150144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696929.701815:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696929.701817:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696929.701818:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696929.701821:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4830 00000100:00080000:0.0:1787696929.701824:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696929.701826:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701828:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701832:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696929.701835:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696929.701842:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701848:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701852:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696929.701859:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696929.701864:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696929.701868:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696929.701880:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126150272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696929.701891:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126150272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696929.701900:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696929.701907:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696929.701920:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696929.701929:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4830 00000100:00080000:0.0:1787696929.701937:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696929.701942:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701956:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696929.701961:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696929.701966:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.701969:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.701971:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696929.701973:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696929.701978:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696929.701983:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696929.701993:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126150400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696929.702029:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126150400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696929.702041:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696929.702051:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696929.702054:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696929.702063:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4830 00000100:00080000:0.0:1787696929.702071:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696929.702078:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.702087:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.702092:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696929.702097:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696929.702112:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696929.702120:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696929.702128:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696929.702134:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696929.702143:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696929.702151:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696929.702164:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126150528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696929.702181:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126150528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696929.702207:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4840) 00000100:00100000:3.0:1787696930.213505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696931.237517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696931.621520:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299505007, at 4299505024 00000100:00100000:3.0:1787696932.261672:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696933.285533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696933.285566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696934.309526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696934.309526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696934.821579:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696934.821598:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696934.821605:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696934.821616:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4835 00000100:00080000:3.0:1787696934.821628:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696934.821632:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.821636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.821640:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696934.821646:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696934.821649:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.821653:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.821656:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696934.821660:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696934.821675:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696934.821736:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126150656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696934.821752:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126150656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696934.821776:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696934.821792:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696934.821795:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696934.821802:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4835 00000100:00080000:3.0:1787696934.821807:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696934.821810:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.821814:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.821817:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696934.821820:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696934.821827:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.821835:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.821842:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696934.821847:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696934.821860:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696934.821868:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696934.821883:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126150784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696934.821904:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126150784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696934.821916:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696934.821919:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696934.821922:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696934.821929:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4835 00000100:00080000:3.0:1787696934.821936:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696934.821940:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.821944:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.821956:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696934.821961:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696934.821967:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.821970:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.821973:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696934.821977:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696934.821985:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696934.821995:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696934.822011:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126150912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696934.822043:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126150912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696934.822062:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696934.822067:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696934.822070:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696934.822080:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4835 00000100:00080000:3.0:1787696934.822088:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696934.822098:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.822106:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.822113:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696934.822118:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696934.822125:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.822128:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.822130:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696934.822133:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696934.822138:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696934.822154:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696934.822170:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126151040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696934.822207:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126151040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696934.822222:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696934.822229:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696934.822233:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696934.822241:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4835 00000100:00080000:3.0:1787696934.822248:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696934.822256:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.822265:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.822269:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696934.822274:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696934.822282:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696934.822289:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696934.822293:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696934.822299:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696934.822306:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696934.822315:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696934.822328:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126151168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696934.822349:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126151168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696934.822369:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4845) 00000100:00100000:0.0:1787696935.333507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696935.333507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696936.357523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696936.357523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696936.741495:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299510127, at 4299510144 00000100:00100000:0.0:1787696937.381503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696937.381507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696938.405522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696938.405522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696939.429506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696939.429507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696939.941518:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696939.941527:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696939.941530:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696939.941538:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4840 00000100:00080000:3.0:1787696939.941548:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696939.941551:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941555:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941558:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696939.941562:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696939.941565:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941567:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941569:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696939.941571:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941583:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696939.941625:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126151296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696939.941641:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126151296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696939.941665:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696939.941668:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696939.941670:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696939.941686:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4840 00000100:00080000:3.0:1787696939.941689:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696939.941692:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941694:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941696:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696939.941699:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696939.941701:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941704:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941706:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696939.941708:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696939.941716:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696939.941721:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696939.941734:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126151424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696939.941753:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126151424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696939.941761:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696939.941764:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696939.941765:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696939.941768:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4840 00000100:00080000:3.0:1787696939.941772:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696939.941779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941785:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941789:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696939.941793:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696939.941800:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941810:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696939.941815:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696939.941820:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696939.941824:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696939.941831:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126151552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696939.941838:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126151552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696939.941844:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696939.941846:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696939.941848:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696939.941850:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4840 00000100:00080000:3.0:1787696939.941854:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696939.941855:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941857:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941859:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696939.941863:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696939.941868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941872:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941873:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696939.941875:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696939.941879:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696939.941885:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696939.941895:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126151680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696939.941907:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126151680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696939.941913:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696939.941915:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696939.941916:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696939.941919:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4840 00000100:00080000:3.0:1787696939.941923:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696939.941929:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941936:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941941:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696939.941944:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696939.941946:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696939.941948:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696939.941950:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696939.941952:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696939.941959:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696939.941963:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696939.941973:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126151808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696939.941984:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126151808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696939.942015:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4851) 00000100:00100000:0.0:1787696940.453546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696941.477609:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696941.861591:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299515247, at 4299515264 00000100:00100000:0.0:1787696942.501578:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696943.525570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696944.549501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787696945.061597:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696945.061607:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696945.061612:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696945.061623:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4846 00000100:00080000:0.0:1787696945.061637:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696945.061662:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061666:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061670:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696945.061675:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696945.061678:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061680:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061682:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696945.061685:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061699:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696945.061728:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126151936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696945.061742:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126151936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696945.061769:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696945.061773:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696945.061774:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696945.061778:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4846 00000100:00080000:0.0:1787696945.061781:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696945.061784:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061786:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061787:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696945.061789:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696945.061791:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061793:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061795:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696945.061797:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696945.061806:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696945.061814:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696945.061823:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126152064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696945.061836:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126152064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696945.061860:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696945.061864:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696945.061866:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696945.061869:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4846 00000100:00080000:0.0:1787696945.061872:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696945.061874:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061876:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061878:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696945.061880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696945.061882:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061885:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696945.061887:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696945.061892:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696945.061897:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696945.061908:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126152192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696945.061921:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126152192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696945.061933:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696945.061941:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696945.061943:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696945.061947:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4846 00000100:00080000:0.0:1787696945.061957:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696945.061960:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061962:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061964:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696945.061966:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696945.061968:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.061969:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.061971:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696945.061974:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696945.061984:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696945.061991:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696945.062005:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126152320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696945.062023:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126152320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696945.062034:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696945.062037:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696945.062038:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696945.062041:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4846 00000100:00080000:0.0:1787696945.062044:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696945.062047:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.062049:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.062056:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696945.062058:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696945.062061:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696945.062065:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696945.062070:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696945.062075:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696945.062083:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696945.062088:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696945.062099:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126152448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696945.062112:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126152448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696945.062127:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4856) 00000100:00100000:0.0:1787696945.573562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696946.597474:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696946.981534:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299520367, at 4299520384 00000100:00100000:0.0:1787696948.645564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787696950.181597:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696950.181614:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696950.181620:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696950.181634:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4851 00000100:00080000:3.0:1787696950.181649:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696950.181653:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.181658:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.181662:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696950.181668:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696950.181672:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.181674:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.181677:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696950.181680:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696950.181701:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696950.181735:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126152576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696950.181751:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126152576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696950.181779:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696950.181784:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696950.181785:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696950.181790:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4851 00000100:00080000:3.0:1787696950.181793:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696950.181796:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.181799:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.181802:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696950.181822:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696950.181832:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.181845:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.181849:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696950.181858:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696950.181876:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696950.181884:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696950.181907:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126152704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696950.181928:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126152704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696950.181984:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696950.181989:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696950.181992:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696950.181998:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4851 00000100:00080000:3.0:1787696950.182005:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696950.182010:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.182014:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.182017:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696950.182023:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696950.182029:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.182033:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.182041:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696950.182052:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696950.182070:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696950.182077:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696950.182097:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126152832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696950.182133:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126152832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696950.182146:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696950.182153:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696950.182160:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696950.182172:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4851 00000100:00080000:3.0:1787696950.182179:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696950.182185:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.182190:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.182195:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696950.182200:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696950.182206:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.182212:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.182219:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696950.182234:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696950.182245:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696950.182254:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696950.182273:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126152960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696950.182321:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126152960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696950.182335:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696950.182357:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696950.182367:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696950.182375:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4851 00000100:00080000:3.0:1787696950.182384:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696950.182393:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.182399:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.182402:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696950.182408:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696950.182477:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696950.182488:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696950.182500:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696950.182509:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696950.182522:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696950.182535:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696950.182555:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126153088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696950.182600:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126153088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696950.182628:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4861) 00000100:00100000:3.0:1787696950.693546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696951.717531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696952.101497:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299525487, at 4299525504 00000100:00100000:3.0:1787696952.741536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696953.765535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696954.789528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696954.789528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787696955.301494:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696955.301508:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787696955.301513:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696955.301548:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4856 00000100:00080000:3.0:1787696955.301561:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696955.301565:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301569:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301574:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787696955.301580:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787696955.301583:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301585:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301589:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787696955.301592:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301605:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787696955.301636:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126153216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696955.301652:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126153216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696955.301678:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696955.301686:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787696955.301688:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696955.301692:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4856 00000100:00080000:3.0:1787696955.301695:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696955.301698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301701:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301704:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787696955.301707:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787696955.301711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301713:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301727:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787696955.301733:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696955.301747:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696955.301754:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696955.301768:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126153344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696955.301778:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126153344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696955.301792:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696955.301800:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787696955.301802:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696955.301807:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4856 00000100:00080000:3.0:1787696955.301812:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696955.301816:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301819:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301821:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787696955.301823:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787696955.301826:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301828:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301829:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787696955.301831:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696955.301835:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787696955.301840:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787696955.301850:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126153472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696955.301881:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126153472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696955.301887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696955.301890:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787696955.301892:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696955.301896:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4856 00000100:00080000:3.0:1787696955.301900:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696955.301906:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301912:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787696955.301913:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787696955.301919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.301925:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.301930:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787696955.301936:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696955.301945:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696955.301950:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696955.301966:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126153600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696955.301991:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126153600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787696955.302004:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787696955.302007:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787696955.302009:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787696955.302018:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4856 00000100:00080000:3.0:1787696955.302022:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787696955.302026:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.302032:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.302036:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787696955.302040:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787696955.302046:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787696955.302050:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787696955.302052:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787696955.302053:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787696955.302057:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787696955.302062:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787696955.302078:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126153728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787696955.302092:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126153728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787696955.302106:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4866) 00000100:00100000:3.0:1787696955.813550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696955.813559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696956.837493:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696956.837497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696957.221489:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299530607, at 4299530624 00000100:00100000:3.0:1787696957.861522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696957.861523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696958.885517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696958.885518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696959.909507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696959.909507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696960.933539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696960.933648:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696961.957492:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787696962.341505:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299535727, at 4299535744 00000100:00100000:0.0:1787696962.981543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696964.005519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696965.029521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696967.077601:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696967.461542:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299540847, at 4299540864 00000100:00100000:3.0:1787696968.101519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696969.125538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696970.149535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696971.173628:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696972.197571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696972.581539:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299545967, at 4299545984 00000100:00100000:3.0:1787696973.221515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696974.245483:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696975.269531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696976.293544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696976.293581:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696977.317542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696977.317632:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696977.701538:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299551087, at 4299551104 00000100:00100000:0.0:1787696978.341507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696978.341507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696979.365526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696980.389535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696981.413512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696982.437554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696982.821554:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299556207, at 4299556224 00000100:00100000:3.0:1787696983.461560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696984.485553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696985.509517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787696986.021544:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696986.021551:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787696986.021555:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696986.021563:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4887 00000100:00080000:0.0:1787696986.021571:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696986.021574:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021577:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021579:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787696986.021583:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787696986.021585:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021587:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021589:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787696986.021591:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021601:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787696986.021620:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126157056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696986.021631:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126157056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696986.021647:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696986.021649:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787696986.021655:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696986.021658:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4887 00000100:00080000:0.0:1787696986.021660:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696986.021662:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021664:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021677:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787696986.021679:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787696986.021681:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021683:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021685:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787696986.021687:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696986.021694:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696986.021698:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696986.021710:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126157184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696986.021726:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126157184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696986.021733:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696986.021735:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787696986.021737:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696986.021739:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4887 00000100:00080000:0.0:1787696986.021743:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696986.021748:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021755:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021758:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787696986.021762:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787696986.021766:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021768:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021776:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787696986.021777:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696986.021781:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787696986.021789:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787696986.021796:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126157312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696986.021816:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126157312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696986.021826:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696986.021829:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787696986.021830:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696986.021834:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4887 00000100:00080000:0.0:1787696986.021837:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696986.021842:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021848:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021852:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787696986.021856:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787696986.021858:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021860:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021863:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787696986.021866:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696986.021871:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696986.021876:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696986.021888:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126157440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696986.021905:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126157440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787696986.021930:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787696986.021942:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787696986.021944:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787696986.021955:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4887 00000100:00080000:0.0:1787696986.021960:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787696986.021966:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021972:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021976:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787696986.021978:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787696986.021984:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787696986.021987:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787696986.021988:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787696986.021993:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787696986.021997:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787696986.022004:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787696986.022017:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126157568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787696986.022036:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126157568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787696986.022053:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4897) 00000100:00100000:3.0:1787696986.533515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696987.557533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696987.941529:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299561327, at 4299561344 00000100:00100000:3.0:1787696988.581543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696989.605605:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696990.629493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696990.630262:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696991.653639:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696991.653641:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696992.677549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696992.677550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696993.061495:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299566447, at 4299566464 00000100:00100000:0.0:1787696993.701521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696993.701521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000400:00000100:3.0:1787696994.213767:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787696994.213823:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787696994.213842:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696994.213884:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126157696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697016 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696994.213904:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696994.213915:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126157824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697016 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696994.213926:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696994.213937:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126157952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697016 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696994.213950:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696994.213963:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126158080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697016 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787696994.213976:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787696994.213989:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126158208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697016 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787696994.725535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696994.725569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696995.749545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696996.773557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787696997.797539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696997.797632:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787696998.182225:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299571567, at 4299571584 00000100:00100000:0.0:1787696998.821524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787696998.821524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787696999.845537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787696999.845565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697000.869581:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697000.870930:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697001.381588:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697001.381598:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697001.381602:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697001.381613:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4902 00000100:00080000:0.0:1787697001.381624:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697001.381628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.381633:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.381637:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697001.381642:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697001.381645:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.381647:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.381650:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697001.381653:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697001.381666:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697001.381689:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126158976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697001.381704:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126158976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697001.381739:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697001.381743:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697001.381745:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697001.381749:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4902 00000100:00080000:0.0:1787697001.381752:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697001.381770:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.381776:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.381781:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697001.381785:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697001.381792:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.381797:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.381799:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697001.381806:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697001.381817:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697001.381823:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697001.381837:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126159104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697001.381867:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126159104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697001.381877:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697001.381880:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697001.381882:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697001.381885:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4902 00000100:00080000:0.0:1787697001.381890:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697001.381893:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.381895:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.381897:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697001.381900:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697001.381903:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.381905:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.381908:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697001.381919:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697001.381924:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697001.381929:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697001.381944:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126159232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697001.381956:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126159232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697001.381964:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697001.381969:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697001.381973:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697001.381982:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4902 00000100:00080000:0.0:1787697001.381988:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697001.381993:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.382011:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.382017:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697001.382021:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697001.382028:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.382032:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.382036:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697001.382043:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697001.382050:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697001.382058:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697001.382072:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126159360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697001.382087:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126159360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697001.382103:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697001.382112:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697001.382116:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697001.382125:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4902 00000100:00080000:0.0:1787697001.382138:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697001.382144:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.382151:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.382156:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697001.382160:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697001.382166:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697001.382174:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697001.382177:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697001.382180:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697001.382185:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697001.382192:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697001.382213:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126159488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697001.382229:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126159488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697001.382243:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4912) 00000100:00100000:0.0:1787697001.893575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697002.917507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697002.917547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697003.301565:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299576687, at 4299576704 00000100:00100000:3.0:1787697003.941544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697003.941546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697004.965523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697004.965523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697005.989534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697005.989632:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697006.501518:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697006.501528:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697006.501532:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697006.501541:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4907 00000100:00080000:3.0:1787697006.501550:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697006.501553:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501557:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501560:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697006.501566:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697006.501569:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501571:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501580:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697006.501583:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501596:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697006.501619:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126159616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697006.501634:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126159616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697006.501655:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697006.501660:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697006.501661:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697006.501665:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4907 00000100:00080000:3.0:1787697006.501668:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697006.501672:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501675:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501677:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697006.501693:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697006.501700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501703:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501707:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697006.501713:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697006.501724:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697006.501731:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697006.501743:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126159744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697006.501754:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126159744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697006.501771:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697006.501778:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697006.501785:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697006.501790:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4907 00000100:00080000:3.0:1787697006.501796:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697006.501799:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501804:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501808:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697006.501811:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697006.501815:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501820:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501824:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697006.501830:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697006.501836:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697006.501841:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697006.501860:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126159872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697006.501879:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126159872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697006.501907:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697006.501914:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697006.501919:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697006.501928:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4907 00000100:00080000:3.0:1787697006.501934:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697006.501940:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501948:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501951:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697006.501955:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697006.501958:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.501961:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.501963:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697006.501966:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697006.501972:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697006.501979:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697006.501991:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126160000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697006.502008:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126160000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697006.502038:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697006.502042:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697006.502052:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697006.502059:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4907 00000100:00080000:3.0:1787697006.502066:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697006.502072:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.502078:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.502080:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697006.502083:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697006.502090:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697006.502095:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697006.502100:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697006.502105:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697006.502111:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697006.502119:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697006.502134:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126160128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697006.502163:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126160128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697006.502187:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4917) 00000100:00100000:0.0:1787697007.013541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697007.013648:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697008.037528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697008.037528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697008.421520:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299581807, at 4299581824 00000100:00100000:3.0:1787697009.061569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697009.061572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697010.085514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697010.085622:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697011.109524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697011.109525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697012.133570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697012.133571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697013.157549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697013.157551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697013.541589:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299586927, at 4299586944 00000100:00100000:0.0:1787697014.181576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697014.181576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697015.205523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697015.205558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697016.229554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697016.229554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697016.741569:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697016.741584:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697016.741590:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697016.741604:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4917 00000100:00080000:0.0:1787697016.741616:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697016.741619:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741623:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741626:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697016.741631:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697016.741634:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741639:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697016.741641:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741658:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697016.741688:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126160896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697016.741703:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126160896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697016.741726:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697016.741752:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697016.741754:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697016.741760:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4917 00000100:00080000:0.0:1787697016.741765:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697016.741768:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741771:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741773:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697016.741776:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697016.741779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741782:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741785:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697016.741787:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697016.741798:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697016.741803:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697016.741815:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126161024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697016.741826:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126161024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697016.741835:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697016.741838:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697016.741840:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697016.741843:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4917 00000100:00080000:0.0:1787697016.741854:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697016.741857:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741860:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741868:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697016.741871:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697016.741874:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741877:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741879:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697016.741881:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697016.741885:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697016.741889:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697016.741899:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126161152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697016.741908:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126161152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697016.741916:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697016.741918:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697016.741920:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697016.741925:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4917 00000100:00080000:0.0:1787697016.741929:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697016.741932:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741935:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741937:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697016.741939:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697016.741941:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.741944:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.741946:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697016.741948:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697016.741952:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697016.741962:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697016.741972:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126161280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697016.741983:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126161280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697016.741991:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697016.741995:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697016.741997:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697016.742001:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4917 00000100:00080000:0.0:1787697016.742005:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697016.742008:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.742011:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.742013:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697016.742016:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697016.742019:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697016.742022:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697016.742025:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697016.742028:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697016.742032:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697016.742036:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697016.742044:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126161408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697016.742053:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126161408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697016.742065:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4927) 00000100:00100000:3.0:1787697017.253529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697017.253532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697018.277540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697018.277540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697018.661536:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299592047, at 4299592064 00000100:00100000:0.0:1787697019.301569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697019.301569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697020.325532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697020.325532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697021.349563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697021.349563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697022.373503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697022.373550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697023.397587:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697023.397601:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697023.781512:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299597167, at 4299597184 00000100:00100000:3.0:1787697024.421561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697024.421563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697025.445544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697025.445546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697026.469577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697026.469585:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697027.493496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697027.493499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697028.517498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697028.517505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697028.901512:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299602287, at 4299602304 00000100:00100000:0.0:1787697029.541583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697029.541584:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697030.565501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697030.565502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697031.589558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697031.589558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697032.613530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697032.613530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697033.637585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697033.637586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697034.021545:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299607407, at 4299607424 00000100:00100000:0.0:1787697034.661522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697034.661562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697035.685549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697035.685576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697036.709531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697036.709644:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697037.733538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697037.733538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697038.757567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697038.757578:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697039.141509:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299612527, at 4299612544 00000100:00100000:0.0:1787697039.781520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697039.781520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697040.805563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697040.805564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697041.829494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697041.829512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697042.853560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697042.853560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697043.877496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697043.877517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697044.261535:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299617647, at 4299617664 00000100:00100000:3.0:1787697044.901549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697044.901550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697045.925527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697045.925532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697046.949562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697046.949562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697047.973522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697047.973522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697048.997517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697048.997517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697049.381494:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299622767, at 4299622784 00000100:00100000:0.0:1787697050.021501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697050.021502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697051.045514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697051.045514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697052.069501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697052.069501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697052.581550:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697052.581564:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697052.581569:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697052.581581:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4953 00000100:00080000:0.0:1787697052.581595:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697052.581598:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.581603:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.581606:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697052.581612:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697052.581615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.581618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.581621:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697052.581624:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697052.581642:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697052.581701:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126165376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697052.581718:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126165376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697052.581740:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697052.581744:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697052.581745:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697052.581749:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4953 00000100:00080000:0.0:1787697052.581754:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697052.581757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.581779:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.581782:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697052.581787:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697052.581793:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.581799:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.581804:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697052.581809:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697052.581819:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697052.581825:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697052.581839:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126165504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697052.581855:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126165504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697052.581866:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697052.581869:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697052.581871:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697052.581875:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4953 00000100:00080000:0.0:1787697052.581880:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697052.581887:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.581892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.581894:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697052.581896:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697052.581900:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.581907:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.581913:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697052.581917:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697052.581929:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697052.581934:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697052.581957:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126165632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697052.581990:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126165632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697052.582016:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697052.582023:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697052.582026:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697052.582032:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4953 00000100:00080000:0.0:1787697052.582038:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697052.582043:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.582048:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.582051:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697052.582056:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697052.582060:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.582063:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.582066:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697052.582069:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697052.582075:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697052.582081:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697052.582096:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126165760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697052.582107:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126165760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697052.582146:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697052.582152:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697052.582155:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697052.582164:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4953 00000100:00080000:0.0:1787697052.582171:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697052.582175:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.582178:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.582184:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697052.582188:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697052.582195:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697052.582202:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697052.582207:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697052.582214:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697052.582223:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697052.582232:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697052.582247:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126165888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697052.582274:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126165888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697052.582288:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4963) 00000100:00100000:0.0:1787697053.093523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697053.093605:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697054.117508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697054.117544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697054.501529:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299627887, at 4299627904 00000100:00100000:0.0:1787697055.141576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697056.165527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697057.189530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697058.213586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697059.237505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697059.621499:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299633007, at 4299633024 00000100:00100000:0.0:1787697060.261536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000400:00000100:0.0:1787697060.773778:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787697060.773836:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697060.773857:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697060.773906:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126166016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697082 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697060.773929:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697060.773942:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126166144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697082 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697060.773954:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697060.773968:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126166272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697082 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697060.773988:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697060.773997:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126166400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697082 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697060.774015:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697060.774027:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126166528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697082 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697061.285523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697062.309530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697062.821534:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697062.821546:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697062.821551:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697062.821563:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4963 00000100:00080000:3.0:1787697062.821576:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697062.821580:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821584:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821589:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697062.821595:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697062.821599:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821602:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821623:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697062.821627:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821639:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697062.821669:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126166656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697062.821686:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126166656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697062.821708:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697062.821711:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697062.821712:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697062.821719:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4963 00000100:00080000:3.0:1787697062.821722:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697062.821724:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821728:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697062.821730:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697062.821735:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821737:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821738:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697062.821741:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697062.821749:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697062.821755:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697062.821767:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126166784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697062.821779:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126166784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697062.821806:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697062.821809:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697062.821811:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697062.821816:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4963 00000100:00080000:3.0:1787697062.821823:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697062.821830:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821837:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821842:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697062.821847:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697062.821852:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821854:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821857:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697062.821860:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697062.821866:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697062.821876:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697062.821891:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126166912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697062.821918:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126166912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697062.821941:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697062.821946:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697062.821948:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697062.821954:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4963 00000100:00080000:3.0:1787697062.821958:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697062.821968:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821971:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821974:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697062.821978:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697062.821986:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.821994:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.821996:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697062.821999:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697062.822005:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697062.822012:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697062.822031:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126167040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697062.822052:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126167040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697062.822077:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697062.822081:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697062.822084:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697062.822093:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4963 00000100:00080000:3.0:1787697062.822097:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697062.822105:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.822112:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.822115:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697062.822118:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697062.822124:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697062.822128:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697062.822131:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697062.822141:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697062.822150:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697062.822159:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697062.822174:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126167168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697062.822195:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126167168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697062.822213:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4973) 00000100:00100000:3.0:1787697063.333536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697064.357513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697064.741691:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299638127, at 4299638144 00000100:00100000:3.0:1787697065.381524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697066.405556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697067.429540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697067.941578:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697067.941588:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697067.941592:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697067.941602:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4968 00000100:00080000:0.0:1787697067.941614:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697067.941616:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941640:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941644:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697067.941650:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697067.941654:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941657:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941660:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697067.941663:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941677:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697067.941707:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126167296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697067.941728:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126167296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697067.941746:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697067.941748:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697067.941750:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697067.941753:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4968 00000100:00080000:0.0:1787697067.941755:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697067.941757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941759:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941760:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697067.941763:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697067.941764:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941769:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941770:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697067.941772:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697067.941781:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697067.941793:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697067.941807:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126167424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697067.941819:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126167424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697067.941844:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697067.941848:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697067.941851:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697067.941861:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4968 00000100:00080000:0.0:1787697067.941865:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697067.941868:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941871:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941873:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697067.941875:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697067.941879:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941881:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941883:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697067.941886:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697067.941893:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697067.941901:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697067.941916:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126167552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697067.941923:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126167552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697067.941929:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697067.941938:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697067.941939:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697067.941943:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4968 00000100:00080000:0.0:1787697067.941946:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697067.941948:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941950:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941951:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697067.941953:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697067.941956:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.941960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.941963:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697067.941966:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697067.941971:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697067.941974:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697067.941980:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126167680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697067.941991:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126167680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697067.942012:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697067.942014:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697067.942015:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697067.942018:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4968 00000100:00080000:0.0:1787697067.942020:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697067.942021:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.942023:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.942027:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697067.942029:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697067.942032:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697067.942034:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697067.942036:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697067.942038:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697067.942041:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697067.942044:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697067.942056:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126167808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697067.942068:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126167808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697067.942091:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4979) 00000100:00100000:3.0:1787697068.453527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697069.477529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697069.477540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697069.861543:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299643247, at 4299643264 00000100:00100000:3.0:1787697070.501726:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697071.525515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697072.549691:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697073.061556:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.061565:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697073.061570:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.061581:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4974 00000100:00080000:3.0:1787697073.061594:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697073.061596:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061600:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061603:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697073.061630:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697073.061633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061635:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061638:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697073.061640:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061653:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697073.061679:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126167936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697073.061691:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126167936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697073.061711:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.061714:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697073.061715:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.061718:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4974 00000100:00080000:3.0:1787697073.061721:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697073.061723:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061725:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061726:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697073.061728:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697073.061731:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061733:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061734:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697073.061736:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697073.061750:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697073.061755:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697073.061764:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126168064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697073.061784:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126168064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697073.061792:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.061794:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697073.061796:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.061799:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4974 00000100:00080000:3.0:1787697073.061801:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697073.061806:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061811:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061822:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697073.061824:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697073.061827:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061829:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061830:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697073.061832:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697073.061835:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697073.061842:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697073.061849:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126168192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697073.061856:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126168192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697073.061862:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.061867:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697073.061873:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.061878:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4974 00000100:00080000:3.0:1787697073.061882:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697073.061886:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061890:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061893:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697073.061896:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697073.061901:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061905:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061909:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697073.061910:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697073.061914:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697073.061916:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697073.061924:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126168320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697073.061930:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126168320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697073.061935:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697073.061937:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697073.061939:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697073.061943:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4974 00000100:00080000:3.0:1787697073.061948:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697073.061952:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061956:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061959:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697073.061961:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697073.061971:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697073.061976:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697073.061980:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697073.061985:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697073.061989:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697073.061995:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697073.062009:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126168448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697073.062044:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126168448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697073.062064:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4984) 00000100:00100000:3.0:1787697073.573539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697074.597597:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697074.981538:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299648367, at 4299648384 00000100:00100000:3.0:1787697075.621576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697076.645538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697077.669515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697078.181517:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697078.181528:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697078.181533:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697078.181544:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4979 00000100:00080000:3.0:1787697078.181554:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697078.181557:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181560:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181563:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697078.181567:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697078.181570:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181589:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181591:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697078.181594:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181604:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697078.181625:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126168576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697078.181638:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126168576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697078.181661:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697078.181664:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697078.181666:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697078.181669:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4979 00000100:00080000:3.0:1787697078.181672:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697078.181674:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181676:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181678:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697078.181680:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697078.181682:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181683:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181685:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697078.181687:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697078.181695:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697078.181698:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697078.181706:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126168704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697078.181718:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126168704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697078.181756:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697078.181761:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697078.181763:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697078.181771:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4979 00000100:00080000:3.0:1787697078.181774:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697078.181776:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181778:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181780:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697078.181782:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697078.181784:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181786:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181788:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697078.181789:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697078.181794:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697078.181800:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697078.181814:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126168832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697078.181822:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126168832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697078.181828:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697078.181831:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697078.181832:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697078.181834:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4979 00000100:00080000:3.0:1787697078.181837:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697078.181846:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181848:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181850:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697078.181854:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697078.181861:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181864:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181866:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697078.181869:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697078.181875:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697078.181882:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697078.181890:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126168960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697078.181909:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126168960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697078.181915:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697078.181917:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697078.181918:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697078.181920:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4979 00000100:00080000:3.0:1787697078.181924:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697078.181927:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181929:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181931:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697078.181933:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697078.181938:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697078.181940:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697078.181945:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697078.181952:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697078.181956:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697078.181959:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697078.181969:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126169088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697078.181986:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126169088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697078.181996:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4989) 00000100:00100000:3.0:1787697078.693567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697079.717540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697080.101521:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299653487, at 4299653504 00000100:00100000:3.0:1787697080.741521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697081.765540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697082.789497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697083.301565:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697083.301576:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697083.301587:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697083.301599:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4984 00000100:00080000:3.0:1787697083.301611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697083.301615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.301620:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.301623:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697083.301628:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697083.301631:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.301634:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.301637:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697083.301641:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697083.301679:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697083.301708:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126169216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697083.301724:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126169216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697083.301743:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697083.301748:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697083.301750:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697083.301756:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4984 00000100:00080000:3.0:1787697083.301762:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697083.301766:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.301774:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.301781:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697083.301786:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697083.301801:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.301808:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.301811:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697083.301815:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697083.301828:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697083.301836:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697083.301853:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126169344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697083.301868:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126169344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697083.301878:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697083.301890:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697083.301893:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697083.301898:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4984 00000100:00080000:3.0:1787697083.301906:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697083.301913:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.301916:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.301919:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697083.301922:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697083.301927:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.301930:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.301936:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697083.301944:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697083.301956:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697083.301962:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697083.301976:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126169472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697083.301992:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126169472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697083.302017:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697083.302021:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697083.302023:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697083.302032:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4984 00000100:00080000:3.0:1787697083.302041:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697083.302050:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.302058:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.302068:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697083.302074:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697083.302083:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.302091:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.302099:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697083.302107:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697083.302120:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697083.302141:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697083.302162:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126169600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697083.302175:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126169600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697083.302200:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697083.302207:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697083.302212:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697083.302219:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4984 00000100:00080000:3.0:1787697083.302225:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697083.302235:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.302243:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.302248:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697083.302252:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697083.302258:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697083.302263:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697083.302267:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697083.302273:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697083.302281:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697083.302296:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697083.302316:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126169728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697083.302335:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126169728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697083.302364:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4994) 00000100:00100000:0.0:1787697083.813588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697084.837547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697085.221552:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299658607, at 4299658624 00000100:00100000:0.0:1787697085.861559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697086.885526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697087.909530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697088.421580:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697088.421590:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697088.421595:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697088.421605:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4989 00000100:00080000:0.0:1787697088.421615:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697088.421618:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.421622:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.421625:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697088.421630:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697088.421650:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.421652:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.421656:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697088.421659:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697088.421669:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697088.421692:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126169856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697088.421717:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126169856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697088.421739:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697088.421742:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697088.421744:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697088.421748:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4989 00000100:00080000:0.0:1787697088.421752:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697088.421755:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.421757:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.421759:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697088.421762:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697088.421765:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.421768:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.421771:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697088.421774:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697088.421783:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697088.421788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697088.421802:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126169984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697088.421845:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126169984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697088.421859:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697088.421869:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697088.421873:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697088.421882:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4989 00000100:00080000:0.0:1787697088.421898:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697088.421902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.421905:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.421910:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697088.421915:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697088.421919:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.421923:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.421928:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697088.421936:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697088.421946:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697088.421952:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697088.421967:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126170112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697088.421993:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126170112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697088.422027:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697088.422038:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697088.422063:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697088.422073:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4989 00000100:00080000:0.0:1787697088.422083:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697088.422092:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.422098:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.422104:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697088.422108:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697088.422116:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.422123:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.422130:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697088.422137:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697088.422143:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697088.422148:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697088.422163:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126170240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697088.422179:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126170240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697088.422194:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697088.422204:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697088.422208:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697088.422216:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4989 00000100:00080000:0.0:1787697088.422223:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697088.422227:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.422230:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.422233:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697088.422237:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697088.422251:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697088.422256:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697088.422259:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697088.422266:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697088.422275:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697088.422285:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697088.422300:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126170368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697088.422333:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126170368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697088.422356:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (4999) 00000100:00100000:0.0:1787697088.933516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697089.957493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697090.341536:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299663727, at 4299663744 00000100:00100000:0.0:1787697090.981498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697092.005520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697093.029490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697094.053585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697095.077607:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697095.461531:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299668847, at 4299668864 00000100:00100000:0.0:1787697096.101522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697097.125544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697098.149564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697098.661530:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697098.661541:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697098.661545:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697098.661553:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 4999 00000100:00080000:0.0:1787697098.661564:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697098.661568:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661572:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661576:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697098.661598:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697098.661602:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661605:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661609:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697098.661613:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661623:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697098.661644:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126171136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697098.661655:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126171136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697098.661677:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697098.661680:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697098.661682:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697098.661685:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4999 00000100:00080000:0.0:1787697098.661688:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697098.661690:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661692:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661694:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697098.661695:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697098.661697:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661699:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661701:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697098.661703:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697098.661718:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697098.661725:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697098.661744:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126171264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697098.661768:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126171264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697098.661800:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697098.661805:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697098.661808:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697098.661814:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4999 00000100:00080000:0.0:1787697098.661819:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697098.661823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661826:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661838:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697098.661841:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697098.661845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661848:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661851:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697098.661854:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697098.661861:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697098.661891:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697098.661902:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126171392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697098.661918:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126171392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697098.661927:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697098.661931:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697098.661944:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697098.661950:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4999 00000100:00080000:0.0:1787697098.661957:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697098.661964:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661971:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661974:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697098.661977:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697098.661984:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.661989:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.661993:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697098.661998:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697098.662005:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697098.662010:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697098.662022:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126171520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697098.662037:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126171520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697098.662047:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697098.662053:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697098.662056:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697098.662062:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 4999 00000100:00080000:0.0:1787697098.662068:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697098.662072:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.662074:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.662077:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697098.662080:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697098.662091:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697098.662095:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697098.662098:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697098.662102:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697098.662105:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697098.662108:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697098.662115:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126171648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697098.662127:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126171648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697098.662141:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5009) 00000100:00100000:3.0:1787697099.173523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697100.197485:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697100.581540:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299673967, at 4299673984 00000100:00100000:3.0:1787697101.221709:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697102.245544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697103.269500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697103.781551:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697103.781561:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697103.781566:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697103.781577:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5004 00000100:00080000:0.0:1787697103.781589:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697103.781593:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.781598:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.781601:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697103.781625:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697103.781629:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.781639:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.781642:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697103.781645:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697103.781660:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697103.781685:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126171776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697103.781723:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126171776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697103.781746:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697103.781756:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697103.781758:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697103.781764:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5004 00000100:00080000:0.0:1787697103.781772:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697103.781790:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.781794:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.781796:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697103.781800:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697103.781804:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.781807:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.781810:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697103.781813:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697103.781824:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697103.781829:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697103.781843:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126171904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697103.781861:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126171904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697103.781873:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697103.781877:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697103.781880:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697103.781889:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5004 00000100:00080000:0.0:1787697103.781896:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697103.781904:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.781911:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.781915:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697103.781919:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697103.781924:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.781928:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.781930:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697103.781934:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697103.781939:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697103.781944:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697103.781954:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126172032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697103.781973:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126172032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697103.781985:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697103.781993:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697103.781998:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697103.782007:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5004 00000100:00080000:0.0:1787697103.782011:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697103.782014:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.782017:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.782019:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697103.782022:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697103.782025:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.782039:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.782046:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697103.782053:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697103.782059:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697103.782066:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697103.782081:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126172160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697103.782117:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126172160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697103.782138:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697103.782146:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697103.782150:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697103.782161:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5004 00000100:00080000:0.0:1787697103.782167:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697103.782172:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.782178:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.782184:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697103.782189:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697103.782193:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697103.782197:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697103.782203:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697103.782210:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697103.782225:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697103.782232:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697103.782249:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126172288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697103.782275:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126172288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697103.782293:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5014) 00000100:00100000:3.0:1787697104.293522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697105.317583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697105.317595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697105.701509:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299679087, at 4299679104 00000100:00100000:0.0:1787697106.341518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697106.341519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697107.365551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697107.365552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697108.389515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697108.389516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697108.901546:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697108.901563:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697108.901569:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697108.901580:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5009 00000100:00080000:3.0:1787697108.901591:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697108.901595:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.901617:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.901620:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697108.901626:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697108.901630:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.901633:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.901636:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697108.901640:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697108.901652:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697108.901679:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126172416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697108.901735:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126172416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697108.901760:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697108.901765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697108.901774:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697108.901780:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5009 00000100:00080000:3.0:1787697108.901785:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697108.901789:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.901793:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.901796:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697108.901799:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697108.901803:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.901807:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.901810:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697108.901817:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697108.901831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697108.901850:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697108.901867:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126172544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697108.901885:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126172544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697108.901910:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697108.901915:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697108.901918:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697108.901924:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5009 00000100:00080000:3.0:1787697108.901929:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697108.901934:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.901937:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.901939:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697108.901942:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697108.901948:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.901953:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.901957:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697108.901960:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697108.901965:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697108.901971:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697108.901986:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126172672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697108.902018:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126172672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697108.902046:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697108.902058:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697108.902062:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697108.902070:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5009 00000100:00080000:3.0:1787697108.902075:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697108.902083:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.902089:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.902093:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697108.902096:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697108.902100:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.902106:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.902112:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697108.902118:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697108.902126:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697108.902134:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697108.902153:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126172800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697108.902178:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126172800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697108.902200:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697108.902208:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697108.902210:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697108.902215:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5009 00000100:00080000:3.0:1787697108.902219:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697108.902223:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.902230:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.902244:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697108.902248:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697108.902256:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697108.902261:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697108.902263:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697108.902267:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697108.902274:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697108.902285:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697108.902303:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126172928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697108.902321:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126172928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697108.902338:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5020) 00000100:00100000:3.0:1787697109.413522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697109.413524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697110.437514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697110.437514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697110.821538:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299684207, at 4299684224 00000100:00100000:0.0:1787697111.461542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697111.461542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697112.485522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697112.485522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697113.509502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697113.509509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697114.533530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697114.533659:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697115.557518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697115.557518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697115.941643:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299689327, at 4299689344 00000100:00100000:0.0:1787697116.581531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697116.581533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697117.605554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697118.629588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697119.141547:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697119.141555:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697119.141559:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697119.141567:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5020 00000100:00080000:3.0:1787697119.141575:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697119.141591:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141594:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141596:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697119.141600:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697119.141602:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141604:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141606:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697119.141608:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141618:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697119.141636:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126173696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697119.141647:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126173696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697119.141668:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697119.141671:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697119.141673:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697119.141677:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5020 00000100:00080000:3.0:1787697119.141682:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697119.141684:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141687:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141689:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697119.141693:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697119.141696:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141699:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141701:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697119.141704:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697119.141714:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697119.141723:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697119.141732:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126173824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697119.141745:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126173824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697119.141783:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697119.141798:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697119.141804:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697119.141808:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5020 00000100:00080000:3.0:1787697119.141812:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697119.141818:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141832:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141835:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697119.141837:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697119.141839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141842:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141843:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697119.141845:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697119.141848:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697119.141851:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697119.141857:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126173952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697119.141884:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126173952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697119.141911:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697119.141920:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697119.141922:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697119.141926:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5020 00000100:00080000:3.0:1787697119.141929:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697119.141931:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141933:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141935:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697119.141936:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697119.141939:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.141940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.141941:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697119.141943:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697119.141947:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697119.141954:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697119.141964:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126174080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697119.141975:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126174080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697119.141993:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697119.142003:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697119.142006:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697119.142011:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5020 00000100:00080000:3.0:1787697119.142014:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697119.142018:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.142020:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.142040:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697119.142044:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697119.142056:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697119.142062:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697119.142064:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697119.142067:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697119.142071:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697119.142079:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697119.142096:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126174208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697119.142109:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126174208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697119.142166:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5030) 00000100:00100000:0.0:1787697119.653499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697120.677528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697121.061544:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299694447, at 4299694464 00000100:00100000:0.0:1787697121.701514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697122.725528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697123.749548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697124.261544:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697124.261558:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697124.261563:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697124.261576:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5025 00000100:00080000:3.0:1787697124.261589:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697124.261593:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261597:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261602:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697124.261608:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697124.261613:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261615:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261619:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697124.261622:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261653:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697124.261682:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126174336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697124.261700:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126174336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697124.261724:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697124.261728:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697124.261729:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697124.261734:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5025 00000100:00080000:3.0:1787697124.261738:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697124.261741:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261744:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261746:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697124.261749:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697124.261752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261755:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261757:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697124.261760:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697124.261770:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697124.261775:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697124.261787:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126174464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697124.261811:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126174464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697124.261825:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697124.261833:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697124.261835:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697124.261841:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5025 00000100:00080000:3.0:1787697124.261845:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697124.261852:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261863:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697124.261867:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697124.261871:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261873:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261876:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697124.261878:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697124.261884:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697124.261888:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697124.261898:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126174592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697124.261909:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126174592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697124.261918:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697124.261922:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697124.261924:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697124.261928:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5025 00000100:00080000:3.0:1787697124.261932:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697124.261935:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261939:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261946:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697124.261950:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697124.261958:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.261964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.261969:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697124.261972:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697124.261978:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697124.261985:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697124.261999:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126174720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697124.262012:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126174720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697124.262024:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697124.262031:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697124.262034:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697124.262042:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5025 00000100:00080000:3.0:1787697124.262047:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697124.262051:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.262058:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.262063:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697124.262068:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697124.262072:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697124.262075:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697124.262078:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697124.262084:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697124.262089:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697124.262105:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697124.262120:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126174848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697124.262137:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126174848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697124.262153:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5035) 00000100:00100000:0.0:1787697124.773569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697125.797573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697126.181552:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299699567, at 4299699584 00000100:00100000:0.0:1787697126.821541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697126.821553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697127.845529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697127.845529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697128.869537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697128.869537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697129.893520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697129.893520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697130.917529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697130.917529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697131.301535:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299704687, at 4299704704 00000100:00100000:3.0:1787697132.965561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697132.965585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697133.989573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697135.013545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697136.037542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697136.421546:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299709807, at 4299709824 00000100:00100000:0.0:1787697137.061567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697138.085676:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697139.109513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697139.621566:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697139.621580:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697139.621585:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697139.621595:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5040 00000100:00080000:3.0:1787697139.621608:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697139.621610:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621613:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621616:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697139.621622:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697139.621625:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621628:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621659:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697139.621663:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621680:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697139.621710:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126176256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697139.621724:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126176256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697139.621745:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697139.621750:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697139.621751:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697139.621756:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5040 00000100:00080000:3.0:1787697139.621760:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697139.621763:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621766:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621768:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697139.621770:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697139.621773:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621776:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621777:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697139.621780:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697139.621805:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697139.621810:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697139.621820:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126176384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697139.621831:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126176384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697139.621851:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697139.621855:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697139.621856:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697139.621860:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5040 00000100:00080000:3.0:1787697139.621864:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697139.621866:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621869:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621871:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697139.621873:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697139.621876:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621878:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621880:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697139.621882:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697139.621888:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697139.621892:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697139.621902:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126176512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697139.621911:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126176512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697139.621918:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697139.621921:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697139.621923:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697139.621927:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5040 00000100:00080000:3.0:1787697139.621930:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697139.621939:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621942:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621944:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697139.621946:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697139.621949:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.621952:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.621954:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697139.621957:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697139.621962:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697139.621967:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697139.621976:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126176640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697139.621985:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126176640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697139.621994:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697139.621997:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697139.621999:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697139.622002:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5040 00000100:00080000:3.0:1787697139.622006:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697139.622009:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.622012:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.622014:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697139.622016:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697139.622019:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697139.622022:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697139.622024:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697139.622033:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697139.622038:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697139.622043:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697139.622052:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126176768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697139.622063:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126176768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697139.622074:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5050) 00000100:00100000:0.0:1787697140.133543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697141.157577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697141.157640:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697141.541541:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299714928, at 4299714944 00000100:00100000:0.0:1787697142.181547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697142.181547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697143.205545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697143.205552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697144.229542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697144.229543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697144.741678:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697144.741692:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697144.741697:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697144.741710:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5045 00000100:00080000:3.0:1787697144.741723:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697144.741726:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741736:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741740:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697144.741745:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697144.741748:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741749:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741752:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697144.741754:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741771:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697144.741813:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126176896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697144.741825:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126176896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697144.741845:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697144.741848:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697144.741849:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697144.741852:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5045 00000100:00080000:3.0:1787697144.741855:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697144.741856:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741858:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741859:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697144.741861:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697144.741864:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741866:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741868:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697144.741870:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697144.741878:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697144.741881:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697144.741890:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126177024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697144.741905:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126177024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697144.741914:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697144.741920:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697144.741925:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697144.741929:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5045 00000100:00080000:3.0:1787697144.741931:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697144.741933:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741935:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741937:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697144.741939:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697144.741940:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741942:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741943:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697144.741945:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697144.741948:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697144.741954:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697144.741964:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126177152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697144.741971:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126177152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697144.741976:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697144.741978:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697144.741979:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697144.741982:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5045 00000100:00080000:3.0:1787697144.741983:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697144.741985:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741987:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741988:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697144.741990:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697144.741995:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.741997:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.741998:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697144.742000:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697144.742006:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697144.742010:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697144.742016:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126177280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697144.742024:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126177280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697144.742049:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697144.742052:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697144.742054:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697144.742058:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5045 00000100:00080000:3.0:1787697144.742062:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697144.742066:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.742081:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.742086:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697144.742090:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697144.742096:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697144.742101:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697144.742105:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697144.742107:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697144.742111:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697144.742115:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697144.742124:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126177408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697144.742158:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126177408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697144.742179:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5055) 00000100:00100000:3.0:1787697145.253549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697145.253557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697146.277541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697146.277541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697146.661486:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299720047, at 4299720064 00000100:00100000:3.0:1787697147.301541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697147.301552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697148.325713:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697149.349540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697150.373549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697151.397503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697151.781542:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299725167, at 4299725184 00000100:00100000:3.0:1787697152.421533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697153.445545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697154.469515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697154.981563:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697154.981577:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697154.981582:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697154.981598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5055 00000100:00080000:0.0:1787697154.981611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697154.981614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.981618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981623:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697154.981629:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697154.981632:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.981635:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981639:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697154.981643:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981664:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697154.981697:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126178176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697154.981740:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126178176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697154.981759:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697154.981762:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697154.981763:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697154.981767:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5055 00000100:00080000:0.0:1787697154.981770:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697154.981772:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.981773:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981775:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697154.981777:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697154.981779:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.981781:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981782:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697154.981784:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697154.981793:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697154.981798:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697154.981807:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126178304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697154.981814:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126178304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697154.981820:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697154.981822:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697154.981824:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697154.981832:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5055 00000100:00080000:0.0:1787697154.981835:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697154.981837:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.981839:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981840:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697154.981842:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697154.981845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.981846:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981847:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697154.981849:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697154.981854:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697154.981857:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697154.981872:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126178432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697154.981930:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126178432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697154.981948:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697154.981951:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697154.981952:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697154.981957:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5055 00000100:00080000:0.0:1787697154.981964:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697154.981966:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.981969:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.981971:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697154.981974:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697154.981982:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.982010:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.982018:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697154.982021:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697154.982027:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697154.982032:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697154.982049:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126178560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697154.982059:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126178560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697154.982073:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697154.982077:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697154.982078:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697154.982083:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5055 00000100:00080000:0.0:1787697154.982087:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697154.982092:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.982097:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.982103:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697154.982109:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697154.982121:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697154.982132:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697154.982143:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697154.982149:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697154.982155:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697154.982161:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697154.982175:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126178688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697154.982200:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126178688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697154.982240:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5066) 00000100:00100000:3.0:1787697155.493533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697155.493562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697156.517514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697156.517597:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697156.901493:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299730287, at 4299730304 00000100:00100000:0.0:1787697157.541546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697158.565553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697159.589578:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697160.101538:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697160.101551:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697160.101557:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697160.101568:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5061 00000100:00080000:3.0:1787697160.101581:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697160.101585:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.101589:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.101593:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697160.101598:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697160.101625:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.101628:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.101631:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697160.101635:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697160.101661:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697160.101693:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126178816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697160.101711:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126178816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.101740:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697160.101744:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697160.101746:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697160.101750:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5061 00000100:00080000:3.0:1787697160.101755:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697160.101757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.101760:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.101763:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697160.101766:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697160.101769:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.101772:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.101775:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697160.101778:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697160.101789:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697160.101795:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697160.101821:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126178944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697160.101859:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126178944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.101888:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697160.101893:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697160.101896:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697160.101902:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5061 00000100:00080000:3.0:1787697160.101907:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697160.101919:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.101934:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.101938:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697160.101942:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697160.101946:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.101949:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.101952:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697160.101956:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697160.101964:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697160.101974:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697160.101992:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126179072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697160.102006:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126179072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.102067:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697160.102093:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697160.102109:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697160.102123:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5061 00000100:00080000:3.0:1787697160.102139:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697160.102151:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.102162:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.102170:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697160.102180:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697160.102189:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.102200:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.102215:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697160.102223:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697160.102232:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697160.102240:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697160.102272:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126179200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697160.102301:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126179200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697160.102317:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697160.102323:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697160.102328:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697160.102345:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5061 00000100:00080000:3.0:1787697160.102354:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697160.102366:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.102382:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.102389:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697160.102394:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697160.102412:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697160.102461:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697160.102464:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697160.102467:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697160.102475:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697160.102487:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697160.102509:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126179328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697160.102529:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126179328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697160.102551:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5071) 00000100:00100000:0.0:1787697160.613639:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697160.614144:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697161.637582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697162.021521:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299735407, at 4299735424 00000100:00100000:3.0:1787697162.661568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697162.661603:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697163.685494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697163.685495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697164.709539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697164.709539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697165.733537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697165.733537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697166.757518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697166.757519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697167.141573:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299740527, at 4299740544 00000100:00100000:0.0:1787697167.781555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697167.781563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697168.805491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697168.805491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697169.829564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697169.829565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697170.853576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697170.853579:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697171.877559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697171.877561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697172.261539:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299745647, at 4299745664 00000100:00100000:0.0:1787697172.901538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697172.901538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697173.925539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697173.925539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697174.949565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697174.949690:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697175.973540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697175.973666:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697176.997555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697177.381545:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299750767, at 4299750784 00000100:00100000:3.0:1787697178.021583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697179.045541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697180.069590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697181.093504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697182.117536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697182.501538:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299755887, at 4299755904 00000100:00100000:3.0:1787697183.141582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697184.165520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697185.189640:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697186.213504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697187.237586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697187.621552:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299761007, at 4299761024 00000100:00100000:3.0:1787697188.261658:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697189.285510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697190.309504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697190.821550:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697190.821558:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697190.821561:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697190.821569:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5091 00000100:00080000:0.0:1787697190.821594:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697190.821597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821602:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697190.821606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697190.821608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821612:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697190.821614:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821623:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697190.821643:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126182656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697190.821653:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126182656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697190.821672:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697190.821674:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697190.821676:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697190.821678:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5091 00000100:00080000:0.0:1787697190.821680:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697190.821682:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821683:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821685:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697190.821687:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697190.821689:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821691:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821692:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697190.821694:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697190.821705:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697190.821709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697190.821720:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126182784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697190.821736:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126182784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697190.821748:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697190.821753:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697190.821755:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697190.821759:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5091 00000100:00080000:0.0:1787697190.821765:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697190.821768:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821770:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821773:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697190.821777:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697190.821784:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821790:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821793:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697190.821795:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697190.821800:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697190.821806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697190.821816:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126182912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697190.821823:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126182912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697190.821837:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697190.821839:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697190.821841:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697190.821844:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5091 00000100:00080000:0.0:1787697190.821847:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697190.821849:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821850:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821852:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697190.821853:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697190.821856:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821861:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697190.821863:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697190.821869:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697190.821874:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697190.821886:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126183040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697190.821919:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126183040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697190.821947:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697190.821952:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697190.821954:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697190.821962:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5091 00000100:00080000:0.0:1787697190.821968:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697190.821984:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.821988:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.821991:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697190.821995:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697190.822001:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697190.822007:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697190.822010:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697190.822013:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697190.822018:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697190.822024:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697190.822034:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126183168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697190.822048:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126183168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697190.822061:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5101) 00000100:00100000:3.0:1787697191.333752:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697192.357538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697192.741519:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299766127, at 4299766144 00000100:00100000:3.0:1787697193.381509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697194.405528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697195.429506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697196.453528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697197.477503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697197.861572:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299771247, at 4299771264 00000100:00100000:3.0:1787697198.501518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697199.525601:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697200.549549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697201.061576:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697201.061582:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697201.061586:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697201.061593:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5102 00000100:00080000:0.0:1787697201.061602:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697201.061604:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061607:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061609:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697201.061613:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697201.061615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061617:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061619:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697201.061636:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061646:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697201.061664:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126183936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697201.061675:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126183936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697201.061696:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697201.061698:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697201.061699:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697201.061701:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5102 00000100:00080000:0.0:1787697201.061704:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697201.061706:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061707:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061709:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697201.061711:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697201.061712:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061715:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697201.061716:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697201.061723:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697201.061726:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697201.061732:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126184064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697201.061739:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126184064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697201.061760:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697201.061763:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697201.061764:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697201.061770:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5102 00000100:00080000:0.0:1787697201.061775:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697201.061776:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061779:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697201.061781:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697201.061787:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061788:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061790:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697201.061795:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697201.061802:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697201.061807:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697201.061817:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126184192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697201.061823:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126184192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697201.061829:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697201.061831:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697201.061832:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697201.061835:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5102 00000100:00080000:0.0:1787697201.061838:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697201.061847:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061851:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697201.061853:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697201.061858:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061867:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697201.061870:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697201.061874:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697201.061877:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697201.061884:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126184320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697201.061893:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126184320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697201.061920:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697201.061922:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697201.061923:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697201.061927:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5102 00000100:00080000:0.0:1787697201.061931:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697201.061936:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061941:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061943:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697201.061947:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697201.061958:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697201.061964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697201.061967:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697201.061977:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697201.061984:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697201.061989:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697201.061999:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126184448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697201.062019:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126184448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697201.062027:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5112) 00000100:00100000:3.0:1787697201.573558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697202.597536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697202.981480:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299776367, at 4299776384 00000100:00100000:3.0:1787697203.621525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697204.645520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697205.669504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697206.693565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697207.717544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697208.101527:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299781487, at 4299781504 00000100:00100000:3.0:1787697208.741524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697209.765540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697209.766034:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697210.789529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697210.789545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697211.813520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697211.813611:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697212.837504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697212.837508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697213.221523:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299786607, at 4299786624 00000100:00100000:0.0:1787697213.861523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697213.861523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697214.885500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697214.885500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697215.909483:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697215.909483:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697216.933491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697216.933513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697217.957539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697217.957557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697218.341523:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299791727, at 4299791744 00000100:00100000:0.0:1787697218.981515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697218.981515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697220.005546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697221.029527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697222.053534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697223.077545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697223.461518:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299796847, at 4299796864 00000100:00100000:0.0:1787697224.101508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697225.125503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697226.149516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697226.661533:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697226.661545:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697226.661549:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697226.661557:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5127 00000100:00080000:3.0:1787697226.661566:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697226.661568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661589:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697226.661592:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697226.661595:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661596:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661599:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697226.661601:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661609:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697226.661628:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126187136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697226.661639:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126187136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697226.661675:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697226.661678:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697226.661679:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697226.661683:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5127 00000100:00080000:3.0:1787697226.661686:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697226.661688:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661690:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661692:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697226.661694:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697226.661697:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661699:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661700:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697226.661705:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697226.661712:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697226.661719:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697226.661738:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126187264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697226.661752:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126187264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697226.661773:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697226.661778:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697226.661779:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697226.661785:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5127 00000100:00080000:3.0:1787697226.661790:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697226.661793:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661796:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661798:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697226.661801:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697226.661804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661807:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661809:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697226.661814:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697226.661821:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697226.661828:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697226.661849:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126187392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697226.661869:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126187392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697226.661876:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697226.661880:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697226.661888:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697226.661896:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5127 00000100:00080000:3.0:1787697226.661904:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697226.661909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661913:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661917:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697226.661921:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697226.661926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.661932:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.661936:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697226.661942:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697226.661950:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697226.661957:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697226.661967:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126187520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697226.661975:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126187520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697226.661994:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697226.661999:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697226.662003:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697226.662008:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5127 00000100:00080000:3.0:1787697226.662012:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697226.662019:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.662028:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.662031:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697226.662034:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697226.662047:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697226.662053:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697226.662056:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697226.662059:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697226.662066:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697226.662071:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697226.662084:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126187648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697226.662101:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126187648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697226.662133:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5137) 00000100:00100000:0.0:1787697227.173500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697227.173539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697228.197517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697228.197518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697228.581510:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299801967, at 4299801984 00000100:00100000:0.0:1787697229.221521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697229.221521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697230.245518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697230.245525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697231.269517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697231.269517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697231.781580:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697231.781605:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697231.781608:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697231.781618:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5132 00000100:00080000:0.0:1787697231.781628:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697231.781632:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781640:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697231.781644:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697231.781657:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781660:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781664:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697231.781667:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781678:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697231.781700:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126187776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697231.781713:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126187776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697231.781729:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697231.781732:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697231.781733:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697231.781736:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5132 00000100:00080000:0.0:1787697231.781739:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697231.781740:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781742:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781743:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697231.781746:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697231.781752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781755:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781756:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697231.781758:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697231.781766:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697231.781770:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697231.781782:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126187904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697231.781797:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126187904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697231.781804:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697231.781806:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697231.781807:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697231.781814:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5132 00000100:00080000:0.0:1787697231.781819:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697231.781823:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781826:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697231.781828:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697231.781833:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781839:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781843:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697231.781848:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697231.781854:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697231.781859:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697231.781865:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126188032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697231.781881:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126188032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697231.781892:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697231.781895:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697231.781897:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697231.781900:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5132 00000100:00080000:0.0:1787697231.781905:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697231.781911:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781919:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781922:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697231.781924:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697231.781927:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.781930:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.781933:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697231.781938:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697231.781949:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697231.781956:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697231.781971:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126188160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697231.781987:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126188160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697231.782021:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697231.782027:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697231.782030:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697231.782041:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5132 00000100:00080000:0.0:1787697231.782048:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697231.782055:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.782064:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.782067:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697231.782069:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697231.782074:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697231.782080:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697231.782085:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697231.782087:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697231.782091:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697231.782097:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697231.782110:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126188288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697231.782121:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126188288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697231.782135:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5142) 00000100:00100000:3.0:1787697232.293507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697232.293508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697233.317499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697233.317503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697233.701518:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299807087, at 4299807104 00000100:00100000:3.0:1787697234.341504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697235.365527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697236.389511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697236.901536:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697236.901545:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697236.901549:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697236.901558:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5137 00000100:00080000:3.0:1787697236.901568:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697236.901588:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901591:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901593:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697236.901597:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697236.901599:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901601:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901603:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697236.901605:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901615:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697236.901634:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126188416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697236.901646:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126188416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697236.901664:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697236.901666:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697236.901667:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697236.901670:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5137 00000100:00080000:3.0:1787697236.901673:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697236.901674:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901675:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901677:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697236.901679:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697236.901680:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901682:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901683:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697236.901684:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697236.901691:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697236.901699:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697236.901706:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126188544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697236.901724:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126188544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697236.901737:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697236.901741:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697236.901743:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697236.901748:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5137 00000100:00080000:3.0:1787697236.901752:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697236.901755:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901758:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901761:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697236.901764:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697236.901770:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901775:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901777:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697236.901779:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697236.901783:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697236.901789:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697236.901799:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126188672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697236.901805:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126188672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697236.901811:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697236.901822:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697236.901825:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697236.901829:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5137 00000100:00080000:3.0:1787697236.901833:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697236.901837:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901841:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901845:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697236.901848:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697236.901854:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901856:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901858:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697236.901860:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697236.901864:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697236.901867:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697236.901875:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126188800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697236.901900:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126188800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697236.901921:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697236.901926:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697236.901928:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697236.901932:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5137 00000100:00080000:3.0:1787697236.901936:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697236.901939:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901944:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901950:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697236.901953:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697236.901962:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697236.901967:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697236.901971:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697236.901976:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697236.901981:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697236.901986:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697236.901995:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126188928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697236.902005:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126188928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697236.902026:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5148) 00000100:00100000:3.0:1787697237.413565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697238.437533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697238.821538:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299812207, at 4299812224 00000100:00100000:3.0:1787697239.461529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697240.485525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697241.509581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697241.509599:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697242.533525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697242.533525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697243.557519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697243.557520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697243.941488:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299817327, at 4299817344 00000100:00100000:0.0:1787697244.581508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697244.581509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697245.605517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697245.605548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697246.629512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697246.629653:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697247.141599:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697247.141611:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697247.141615:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697247.141652:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5148 00000100:00080000:3.0:1787697247.141665:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697247.141670:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.141675:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.141678:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697247.141683:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697247.141685:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.141687:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.141690:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697247.141692:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697247.141705:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697247.141746:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126189696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697247.141771:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126189696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.141798:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697247.141802:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697247.141805:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697247.141810:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5148 00000100:00080000:3.0:1787697247.141814:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697247.141819:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.141823:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.141826:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697247.141829:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697247.141832:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.141835:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.141846:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697247.141853:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697247.141870:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697247.141881:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697247.141900:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126189824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697247.141927:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126189824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.141944:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697247.141948:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697247.141950:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697247.141955:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5148 00000100:00080000:3.0:1787697247.141959:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697247.141964:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.141967:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.141969:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697247.141972:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697247.141982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.141990:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.141993:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697247.141999:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697247.142012:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697247.142021:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697247.142037:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126189952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697247.142069:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126189952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.142100:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697247.142105:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697247.142107:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697247.142119:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5148 00000100:00080000:3.0:1787697247.142136:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697247.142161:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.142172:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.142176:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697247.142183:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697247.142198:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.142208:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.142216:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697247.142228:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697247.142239:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697247.142250:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697247.142263:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126190080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697247.142279:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126190080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697247.142333:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697247.142349:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697247.142356:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697247.142370:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5148 00000100:00080000:3.0:1787697247.142378:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697247.142384:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.142394:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.142404:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697247.142411:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697247.142462:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697247.142475:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697247.142481:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697247.142492:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697247.142507:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697247.142513:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697247.142529:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126190208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697247.142566:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126190208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697247.142592:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5158) 00000100:00100000:3.0:1787697247.653544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697247.653563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697248.677481:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697248.677491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697249.061526:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299822447, at 4299822464 00000100:00100000:0.0:1787697249.701512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697249.701512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697250.725538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697250.725551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697251.749538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697251.749538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697252.773518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697252.773518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697253.797516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697253.797516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697254.181492:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299827567, at 4299827584 00000100:00100000:0.0:1787697254.821515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697254.821515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697255.845501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697255.845694:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697256.869519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697256.869520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697257.381524:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697257.381532:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697257.381536:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697257.381543:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5158 00000100:00080000:3.0:1787697257.381552:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697257.381556:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381563:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697257.381568:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697257.381572:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381574:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381578:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697257.381581:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381595:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697257.381623:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126190976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697257.381647:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126190976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697257.381664:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697257.381667:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697257.381668:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697257.381672:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5158 00000100:00080000:3.0:1787697257.381676:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697257.381679:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381696:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381699:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697257.381702:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697257.381706:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381711:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381718:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697257.381725:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697257.381736:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697257.381743:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697257.381756:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126191104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697257.381770:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126191104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697257.381787:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697257.381790:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697257.381792:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697257.381795:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5158 00000100:00080000:3.0:1787697257.381799:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697257.381800:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381802:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381804:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697257.381806:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697257.381808:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381811:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697257.381813:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697257.381822:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697257.381825:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697257.381835:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126191232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697257.381846:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126191232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697257.381855:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697257.381861:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697257.381865:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697257.381871:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5158 00000100:00080000:3.0:1787697257.381878:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697257.381882:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381888:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381892:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697257.381895:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697257.381899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.381906:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.381913:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697257.381928:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697257.381936:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697257.381943:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697257.381956:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126191360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697257.381973:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126191360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697257.382005:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697257.382007:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697257.382009:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697257.382013:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5158 00000100:00080000:3.0:1787697257.382019:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697257.382022:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.382027:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.382032:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697257.382037:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697257.382045:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697257.382050:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697257.382053:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697257.382060:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697257.382071:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697257.382077:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697257.382090:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126191488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697257.382101:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126191488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697257.382120:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5168) 00000100:00100000:0.0:1787697257.893486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697257.893490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697258.917477:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697258.917477:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697259.301504:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299832687, at 4299832704 00000100:00100000:0.0:1787697259.941543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697259.941543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000400:00000100:0.0:1787697260.453719:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787697260.453767:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697260.453786:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697260.453829:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126190976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697282 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697260.453850:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697260.453860:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126191104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697282 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697260.453874:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697260.453884:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126191232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697282 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697260.453898:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697260.453911:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126191360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697282 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697260.453926:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697260.453937:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126191488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697282 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697260.965519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697260.965519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697261.989523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697261.989523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697262.501673:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697262.501687:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697262.501692:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697262.501709:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5163 00000100:00080000:3.0:1787697262.501722:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697262.501725:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.501730:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.501734:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697262.501741:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697262.501744:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.501747:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.501750:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697262.501753:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697262.501769:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697262.501814:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126191616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697262.501832:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126191616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697262.501864:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697262.501868:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697262.501870:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697262.501875:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5163 00000100:00080000:3.0:1787697262.501879:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697262.501882:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.501884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.501907:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697262.501911:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697262.501914:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.501917:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.501919:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697262.501922:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697262.501934:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697262.501939:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697262.501949:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126191744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697262.501959:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126191744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697262.501966:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697262.501969:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697262.501971:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697262.501975:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5163 00000100:00080000:3.0:1787697262.501979:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697262.501982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.501985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.501987:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697262.501990:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697262.501993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.501996:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.501999:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697262.502002:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697262.502007:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697262.502016:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697262.502028:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126191872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697262.502064:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126191872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697262.502077:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697262.502082:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697262.502084:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697262.502088:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5163 00000100:00080000:3.0:1787697262.502092:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697262.502095:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.502098:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.502100:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697262.502103:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697262.502105:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.502108:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.502110:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697262.502112:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697262.502118:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697262.502124:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697262.502138:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126192000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697262.502149:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126192000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697262.502157:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697262.502166:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697262.502168:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697262.502173:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5163 00000100:00080000:3.0:1787697262.502181:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697262.502185:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.502187:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.502189:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697262.502192:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697262.502196:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697262.502201:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697262.502206:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697262.502288:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697262.502295:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697262.502303:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697262.502317:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126192128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697262.502342:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126192128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697262.502359:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5173) 00000100:00100000:0.0:1787697263.013529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697263.013532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697264.037565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697264.037570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697264.421577:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299837807, at 4299837824 00000100:00100000:0.0:1787697265.061554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697265.061555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697266.085508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697266.085607:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697267.109555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697267.109569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697268.133536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697268.135024:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697269.157509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697269.157509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697269.541534:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299842927, at 4299842944 00000100:00100000:3.0:1787697270.181543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697270.181648:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697271.205640:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697271.205703:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697272.229513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697272.229513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697273.253502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697273.253503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697274.277527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697274.277527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697274.661508:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299848047, at 4299848064 00000100:00100000:0.0:1787697275.301513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697275.301523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697276.325494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697276.325500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697277.349551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697277.349552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697278.373576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697278.373577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697279.397540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697279.397540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697279.781540:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299853167, at 4299853184 00000100:00100000:0.0:1787697280.421542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697280.421579:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697281.445543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697281.445543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697282.469527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697282.469549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697283.493510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697283.493510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697284.517539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697284.517542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697284.901517:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299858287, at 4299858304 00000100:00100000:3.0:1787697285.541525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697285.541526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697286.565525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697286.565526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697287.589468:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697287.589487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697288.613570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697288.613649:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697289.637590:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697289.637744:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697290.021542:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299863407, at 4299863424 00000100:00100000:3.0:1787697290.661535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697290.661567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697291.685523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697292.709549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697293.221574:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697293.221589:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697293.221615:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697293.221626:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5194 00000100:00080000:3.0:1787697293.221638:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697293.221640:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.221644:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.221646:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697293.221651:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697293.221660:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.221666:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.221670:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697293.221673:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697293.221690:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697293.221723:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126195456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697293.221739:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126195456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697293.221761:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697293.221765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697293.221767:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697293.221771:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5194 00000100:00080000:3.0:1787697293.221775:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697293.221779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.221782:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.221784:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697293.221788:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697293.221791:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.221794:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.221805:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697293.221813:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697293.221825:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697293.221831:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697293.221844:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126195584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697293.221872:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126195584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697293.221882:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697293.221884:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697293.221886:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697293.221889:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5194 00000100:00080000:3.0:1787697293.221893:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697293.221895:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.221896:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.221897:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697293.221900:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697293.221902:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.221904:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.221906:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697293.221907:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697293.221911:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697293.221915:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697293.221928:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126195712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697293.221965:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126195712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697293.221985:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697293.221989:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697293.221991:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697293.221995:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5194 00000100:00080000:3.0:1787697293.222000:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697293.222003:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.222006:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.222008:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697293.222011:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697293.222014:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.222017:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.222021:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697293.222026:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697293.222034:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697293.222047:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697293.222058:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126195840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697293.222072:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126195840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697293.222082:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697293.222090:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697293.222095:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697293.222109:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5194 00000100:00080000:3.0:1787697293.222118:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697293.222124:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.222128:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.222130:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697293.222132:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697293.222134:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697293.222136:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697293.222138:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697293.222143:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697293.222151:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697293.222157:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697293.222168:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126195968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697293.222184:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126195968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697293.222198:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5204) 00000100:00100000:3.0:1787697293.733553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697294.757530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697295.141554:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299868527, at 4299868544 00000100:00100000:3.0:1787697295.781586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697296.805482:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697297.829573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697298.341536:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697298.341551:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697298.341556:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697298.341570:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5199 00000100:00080000:3.0:1787697298.341604:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697298.341606:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341614:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697298.341619:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697298.341623:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341625:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341628:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697298.341631:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341645:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697298.341679:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126196096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697298.341698:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126196096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697298.341717:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697298.341720:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697298.341721:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697298.341725:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5199 00000100:00080000:3.0:1787697298.341728:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697298.341729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341731:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341733:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697298.341736:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697298.341738:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341740:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341742:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697298.341744:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697298.341763:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697298.341770:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697298.341786:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126196224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697298.341802:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126196224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697298.341809:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697298.341811:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697298.341813:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697298.341816:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5199 00000100:00080000:3.0:1787697298.341821:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697298.341826:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341832:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341835:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697298.341839:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697298.341843:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341844:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341846:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697298.341848:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697298.341852:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697298.341856:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697298.341863:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126196352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697298.341870:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126196352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697298.341888:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697298.341891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697298.341895:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697298.341901:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5199 00000100:00080000:3.0:1787697298.341907:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697298.341910:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341913:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697298.341915:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697298.341917:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.341918:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.341919:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697298.341922:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697298.341928:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697298.341933:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697298.341957:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126196480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697298.341982:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126196480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697298.341999:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697298.342004:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697298.342007:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697298.342015:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5199 00000100:00080000:3.0:1787697298.342022:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697298.342034:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.342041:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.342046:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697298.342050:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697298.342056:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697298.342062:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697298.342065:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697298.342069:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697298.342076:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697298.342082:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697298.342092:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126196608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697298.342099:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126196608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697298.342108:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5209) 00000100:00100000:3.0:1787697298.853532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697298.853588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697299.877558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697299.877558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697300.261504:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299873647, at 4299873664 00000100:00100000:0.0:1787697300.901527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697300.901527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697301.925551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697301.925558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697302.949507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697302.949516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:0.0:1787697303.461543:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697303.461554:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697303.461559:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697303.461568:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5204 00000100:00080000:0.0:1787697303.461579:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697303.461582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461586:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461590:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697303.461594:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697303.461597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461602:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697303.461605:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461619:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697303.461644:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126196736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697303.461658:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126196736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697303.461680:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697303.461684:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697303.461686:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697303.461691:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5204 00000100:00080000:0.0:1787697303.461694:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697303.461698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461701:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461717:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697303.461722:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697303.461729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461741:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697303.461745:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697303.461754:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697303.461760:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697303.461773:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126196864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697303.461792:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126196864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697303.461802:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697303.461806:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697303.461808:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697303.461812:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5204 00000100:00080000:0.0:1787697303.461816:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697303.461822:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461827:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697303.461831:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697303.461834:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461836:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461838:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697303.461842:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697303.461846:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697303.461857:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697303.461871:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126196992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697303.461883:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126196992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697303.461905:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697303.461910:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697303.461914:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697303.461923:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5204 00000100:00080000:0.0:1787697303.461930:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697303.461937:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461945:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461950:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697303.461955:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697303.461963:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.461972:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.461978:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697303.461982:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697303.461990:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697303.461997:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697303.462008:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126197120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697303.462036:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126197120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697303.462047:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697303.462054:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697303.462056:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697303.462061:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5204 00000100:00080000:0.0:1787697303.462064:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697303.462070:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.462075:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.462079:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697303.462083:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697303.462088:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697303.462091:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697303.462093:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697303.462094:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697303.462098:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697303.462102:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697303.462112:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126197248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697303.462121:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126197248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697303.462133:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5214) 00000100:00100000:3.0:1787697303.973473:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697303.973548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697303.973674:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697304.997493:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697304.997501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697305.381539:0:88270:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299878767, at 4299878784 00000100:00100000:0.0:1787697306.021520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697306.021559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697307.045483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697308.069532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697308.581544:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697308.581554:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697308.581558:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697308.581568:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5209 00000100:00080000:3.0:1787697308.581594:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697308.581598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581602:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581605:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697308.581609:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697308.581611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581615:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697308.581617:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581629:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697308.581654:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126197376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697308.581670:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126197376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697308.581694:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697308.581698:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697308.581700:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697308.581704:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5209 00000100:00080000:3.0:1787697308.581708:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697308.581710:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581713:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581715:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697308.581718:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697308.581721:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581726:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697308.581729:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697308.581742:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697308.581746:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697308.581760:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126197504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697308.581787:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126197504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697308.581795:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697308.581798:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697308.581799:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697308.581802:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5209 00000100:00080000:3.0:1787697308.581805:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697308.581808:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581812:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581816:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697308.581819:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697308.581825:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581830:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581832:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697308.581837:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697308.581844:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697308.581849:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697308.581860:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126197632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697308.581870:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126197632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697308.581904:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697308.581910:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697308.581912:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697308.581917:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5209 00000100:00080000:3.0:1787697308.581922:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697308.581926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581933:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697308.581938:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697308.581945:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.581951:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.581953:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697308.581955:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697308.581960:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697308.581965:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697308.581976:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126197760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697308.581999:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126197760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697308.582015:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697308.582024:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697308.582028:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697308.582036:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5209 00000100:00080000:3.0:1787697308.582043:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697308.582054:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.582056:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.582061:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697308.582065:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697308.582071:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697308.582076:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697308.582082:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697308.582085:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697308.582091:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697308.582096:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697308.582108:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126197888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697308.582129:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126197888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697308.582148:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5219) 00000100:00100000:0.0:1787697309.093518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697310.117524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697310.501554:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299883887, at 4299883904 00000100:00100000:0.0:1787697311.141525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697312.165530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697313.189592:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697313.189729:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697313.701528:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697313.701541:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697313.701545:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697313.701556:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5214 00000100:00080000:3.0:1787697313.701566:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697313.701568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701573:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701576:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697313.701581:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697313.701585:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701613:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697313.701622:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701633:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697313.701657:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126198016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697313.701672:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126198016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697313.701695:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697313.701698:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697313.701704:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697313.701708:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5214 00000100:00080000:3.0:1787697313.701712:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697313.701716:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701719:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701721:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697313.701724:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697313.701728:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701731:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701733:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697313.701738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697313.701747:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697313.701754:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697313.701770:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126198144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697313.701785:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126198144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697313.701803:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697313.701807:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697313.701809:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697313.701814:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5214 00000100:00080000:3.0:1787697313.701818:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697313.701821:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701829:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701832:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697313.701835:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697313.701839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701842:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701846:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697313.701852:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697313.701861:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697313.701868:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697313.701879:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126198272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697313.701890:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126198272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697313.701899:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697313.701905:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697313.701908:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697313.701917:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5214 00000100:00080000:3.0:1787697313.701921:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697313.701935:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701939:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701941:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697313.701944:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697313.701950:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.701957:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.701964:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697313.701970:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697313.701977:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697313.701986:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697313.702000:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126198400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697313.702029:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126198400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697313.702042:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697313.702050:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697313.702054:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697313.702061:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5214 00000100:00080000:3.0:1787697313.702069:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697313.702073:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.702076:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.702078:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697313.702081:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697313.702088:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697313.702096:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697313.702100:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697313.702112:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697313.702119:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697313.702125:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697313.702141:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126198528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697313.702168:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126198528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697313.702184:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5224) 00000100:00100000:0.0:1787697314.213579:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697314.213597:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697315.237530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697315.237530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697315.621520:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299889007, at 4299889024 00000100:00100000:0.0:1787697316.261481:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697316.261481:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697317.285532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697317.285533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697318.309531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697318.309531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697319.333490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697319.333492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697320.741566:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299894127, at 4299894144 00000100:00100000:3.0:1787697321.381571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697322.405543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697323.429577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697324.453552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697325.477637:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697325.861538:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299899247, at 4299899264 00000100:00100000:3.0:1787697326.501500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000400:00000100:0.0:1787697327.013681:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787697327.013725:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697327.013741:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697327.013781:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126199296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697348 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697327.013799:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697327.013808:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126199424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697348 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697327.013819:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697327.013827:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126199552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697348 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697327.013836:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697327.013845:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126199680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697348 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697327.013854:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697327.013865:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126199808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697348 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697327.525508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697328.549523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697329.573509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697330.597577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697330.981530:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299904367, at 4299904384 00000100:00100000:3.0:1787697331.621500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697332.645719:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697333.669547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697334.693548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697334.693568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697335.717535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697335.717535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697336.101530:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299909487, at 4299909504 00000100:00100000:0.0:1787697336.741505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697336.741505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697337.765583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697337.765583:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697338.789534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697338.789534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697339.813533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697339.813539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697340.837549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697340.837554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697341.221499:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299914607, at 4299914624 00000100:00100000:0.0:1787697341.861587:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697341.861587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697341.861652:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697342.885556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697342.885556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697343.909499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697343.909499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:2.0:1787697344.421873:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126202112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697344.421888:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697344.421890:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697344.421892:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697344.421895:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5245 00000100:00080000:2.0:1787697344.421897:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697344.421899:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697344.421900:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697344.421902:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697344.421904:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697344.421910:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697344.421913:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697344.421914:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697344.421916:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697344.421920:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697344.421923:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697344.421932:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126202240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697344.421940:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126202240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697344.421969:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697344.421973:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697344.421975:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697344.421980:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5245 00000100:00080000:2.0:1787697344.421984:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697344.421996:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697344.421999:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697344.422005:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697344.422010:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697344.422019:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697344.422024:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697344.422027:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697344.422030:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697344.422038:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697344.422048:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697344.422066:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126202368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697344.422089:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126202368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697344.422109:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5255) 00000100:00100000:0.0:1787697344.933561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697344.933561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697345.957492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697345.957494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697346.341530:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299919727, at 4299919744 00000100:00000040:3.0:1787697346.341530:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299919727, at 4299919744 00000100:00100000:0.0:1787697346.981520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697346.981520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697348.005534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697348.005534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697349.029533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697349.029567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697349.541764:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.541789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126202496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697349.541821:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.541834:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697349.541885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697349.541905:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.541980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697349/real 1787697349] req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542004:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697349.542057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697349.542065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697349.542075:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126202496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697349.542102:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126202496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697349.542119:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697349.542136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126202624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697349.542190:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542199:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697349.542211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697349.542218:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542241:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697349.542245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697349/real 1787697349] req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697349.542263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697349.542265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697349.542267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126202624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697349.542275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126202752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697349.542288:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542292:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697349.542300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697349.542304:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542309:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697349.542319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697349/real 1787697349] req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697349.542346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697349.542349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697349.542353:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126202752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697349.542366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126202880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697349.542387:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542394:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697349.542405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697349.542412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542442:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697349.542449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697349/real 1787697349] req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697349.542476:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697349.542479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697349.542483:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126202880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697349.542495:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126202624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697349.542503:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697349.542508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126202752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697349.542522:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697349.542527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126202880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697349.542544:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697349.542553:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126203008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697349.542583:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542590:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697349.542602:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697349.542609:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697374 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542617:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697349.542621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697349/real 1787697349] req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697349.542638:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697349.542645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697349.542648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697349.542652:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697349.542659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126203008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697349.542665:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126203008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697374 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697349.542672:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697350.053515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697350.053515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697351.077487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697351.077488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697351.461532:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299924847, at 4299924864 00000100:00000040:3.0:1787697351.461542:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299924847, at 4299924864 00000100:00100000:2.0:1787697352.101545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697352.101562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697353.125506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697353.125527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697354.149557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697354.149567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697354.661624:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697354.661635:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697354.661640:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697354.661651:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5255 00000100:00080000:2.0:1787697354.661663:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697354.661667:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.661672:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.661675:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697354.661680:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697354.661683:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.661686:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.661689:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697354.661691:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697354.661708:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697354.661736:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126203136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697354.661751:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126203136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697354.661778:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697354.661787:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697354.661789:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697354.661793:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5255 00000100:00080000:2.0:1787697354.661796:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697354.661816:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.661818:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.661821:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697354.661826:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697354.661830:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.661837:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.661843:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697354.661848:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697354.661860:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697354.661868:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697354.661879:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126203264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697354.661888:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126203264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697354.661897:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697354.661900:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697354.661902:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697354.661909:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5255 00000100:00080000:2.0:1787697354.661915:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697354.661921:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.661927:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.661931:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697354.661936:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697354.661940:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.661942:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.661944:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697354.661958:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697354.661964:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697354.661969:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697354.661978:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126203392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697354.661989:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126203392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697354.661996:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697354.662000:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697354.662003:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697354.662010:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5255 00000100:00080000:2.0:1787697354.662015:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697354.662019:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.662022:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.662024:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697354.662027:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697354.662033:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.662040:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.662045:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697354.662052:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697354.662057:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697354.662060:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697354.662070:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126203520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697354.662079:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126203520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697354.662112:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697354.662120:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697354.662123:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697354.662128:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5255 00000100:00080000:2.0:1787697354.662133:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697354.662140:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.662148:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.662153:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697354.662158:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697354.662163:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697354.662166:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697354.662168:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697354.662171:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697354.662180:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697354.662187:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697354.662203:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126203648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697354.662238:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126203648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697354.662251:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5265) 00000100:00100000:3.0:1787697355.173558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697355.173575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697356.197562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697356.197564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697356.197574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697356.581583:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299929967, at 4299929984 00000100:00000040:3.0:1787697356.581583:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299929967, at 4299929984 00000100:00100000:2.0:1787697357.221569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697357.221570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697357.221570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697358.245564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697358.245564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697358.245564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697359.269533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697359.269533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697359.269538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697359.781555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697359.781569:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697359.781574:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697359.781588:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5260 00000100:00080000:2.0:1787697359.781602:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697359.781606:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.781611:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.781616:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697359.781634:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697359.781639:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.781642:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.781645:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697359.781649:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697359.781667:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697359.781700:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126203776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697359.781718:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126203776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697359.781749:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697359.781753:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697359.781762:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697359.781767:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5260 00000100:00080000:2.0:1787697359.781772:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697359.781775:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.781778:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.781800:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697359.781803:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697359.781815:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.781822:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.781826:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697359.781838:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697359.781851:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697359.781858:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697359.781871:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126203904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697359.781895:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126203904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697359.781908:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697359.781912:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697359.781914:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697359.781919:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5260 00000100:00080000:2.0:1787697359.781924:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697359.781927:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.781930:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.781932:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697359.781935:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697359.781941:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.781946:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.781949:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697359.781952:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697359.781958:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697359.781973:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697359.781989:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126204032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697359.782003:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126204032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697359.782040:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697359.782046:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697359.782049:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697359.782059:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5260 00000100:00080000:2.0:1787697359.782066:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697359.782073:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.782079:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.782084:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697359.782089:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697359.782098:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.782106:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.782111:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697359.782120:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697359.782127:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697359.782134:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697359.782148:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126204160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697359.782168:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126204160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697359.782192:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697359.782205:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697359.782209:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697359.782217:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5260 00000100:00080000:2.0:1787697359.782224:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697359.782231:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.782237:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.782242:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697359.782246:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697359.782251:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697359.782254:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697359.782260:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697359.782266:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697359.782276:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697359.782284:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697359.782301:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126204288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697359.782337:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126204288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697359.782377:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5270) 00000100:00100000:3.0:1787697360.293565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697360.293566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697360.294144:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697361.317486:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697361.317488:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697361.317500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697361.701523:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299935087, at 4299935104 00000100:00000040:2.0:1787697361.701524:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299935087, at 4299935104 00000100:00100000:0.0:1787697362.341509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697362.341509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697362.341509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697363.365489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697363.365495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697363.365515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697364.389503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697364.389503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697364.389524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:2.0:1787697364.901735:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.901756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126204416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697364.901801:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.901818:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697364.901856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697364.901877:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.901918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697364/real 1787697364] req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.901945:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.901957:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697364.901967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697364.901975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697364.901983:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.901998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126204416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697364.902020:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126204416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697364.902039:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697364.902060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697364.902108:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902115:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697364.902129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697364.902139:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902151:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697364.902159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697364/real 1787697364] req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697364.902185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697364.902188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697364.902191:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697364.902202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697364.902217:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902228:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697364.902238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697364.902243:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902251:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697364.902255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697364/real 1787697364] req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902287:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697364.902294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697364.902296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697364.902299:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697364.902311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697364.902336:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902343:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697364.902358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697364.902364:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902372:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697364.902377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697364/real 1787697364] req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697364.902399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697364.902401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697364.902404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697364.902447:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126204544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697364.902455:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697364.902460:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126204672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697364.902466:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697364.902470:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126204800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697364.902476:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697364.902489:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697364.902508:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902514:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697364.902525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697364.902531:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697389 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902538:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697364.902543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697364/real 1787697364] req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902552:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697364.902559:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697364.902565:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697364.902567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697364.902570:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697364.902577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126204928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697364.902581:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126204928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697389 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697364.902597:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697365.413487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697365.413487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697365.413501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697366.437545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697366.437545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697366.437545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697366.821507:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299940207, at 4299940224 00000100:00000040:3.0:1787697366.821516:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299940207, at 4299940224 00000100:00100000:0.0:1787697367.461555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697367.461654:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697367.461724:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697368.485538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697368.485538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697368.485548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697369.509561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697369.509561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697369.509568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697370.021880:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697370.021890:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697370.021894:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697370.021904:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5271 00000100:00080000:3.0:1787697370.021914:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697370.021918:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.021922:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.021925:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697370.021930:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697370.021933:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.021936:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.021939:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697370.021942:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697370.021955:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697370.021992:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697370.022007:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.022028:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697370.022032:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697370.022034:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697370.022037:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5271 00000100:00080000:3.0:1787697370.022041:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697370.022043:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022046:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022048:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697370.022050:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697370.022053:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022056:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022058:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697370.022061:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697370.022070:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697370.022074:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697370.022083:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697370.022092:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.022099:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697370.022101:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697370.022106:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697370.022111:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5271 00000100:00080000:3.0:1787697370.022113:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697370.022116:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022118:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022121:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697370.022123:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697370.022137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697370.022137:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022150:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022159:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697370.022169:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697370.022190:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697370.022199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126205056:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787697370.022200:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697370.022225:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697370.022241:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.022254:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697370.022262:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.022263:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697370.022269:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697370.022289:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697370.022291:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5271 00000100:00080000:3.0:1787697370.022298:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697370.022301:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022304:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022305:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697370.022308:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697370.022315:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022318:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022320:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697370.022322:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697370.022328:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697370.022333:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697370.022345:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697370.022353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022368:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.022380:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697370.022384:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:2.0:1787697370.022388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697370.022389:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697370.022405:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5271 00000100:00080000:3.0:1787697370.022416:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697370.022467:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022472:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022476:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697370.022479:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697370.022482:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697370.022485:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697370.022487:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697370.022490:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697370.022497:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697370.022503:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697370.022516:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697370.022537:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.022538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697370.022622:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5281) 00000100:00100000:2.0:1787697370.022633:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.022648:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697370.022659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697370.022664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697370.022689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.022697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126205056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697370.022705:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.022715:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697370.022735:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.022743:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697370.022759:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697370.022767:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.022779:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697370.022785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.022797:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.022806:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697370.022814:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697370.022817:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697370.022822:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.022843:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697370.022850:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126205056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697370.022863:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697370.022875:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126205184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697370.022883:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697370.022901:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.022910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697370.022926:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.022933:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697370.022947:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697370.022955:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.022965:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697370.022971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.022983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.022992:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697370.023007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697370.023011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697370.023016:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.023024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697370.023030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.023040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697370.023057:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.023064:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697370.023077:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697370.023085:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.023095:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697370.023101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.023112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.023121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697370.023129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697370.023139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697370.023144:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.023151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697370.023158:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.023167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697370.023195:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.023202:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697370.023216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697370.023223:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697395 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.023233:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697370.023239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697370/real 1787697370] req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.023251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697370.023260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697370.023267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697370.023270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697370.023274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697370.023288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697370.023293:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126205312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697370.023302:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697370.023308:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126205440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697370.023315:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697370.023321:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126205568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697395 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697370.023329:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697370.533522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697370.533575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697371.557705:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697371.941566:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299945328, at 4299945344 00000100:00000040:3.0:1787697371.941566:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299945328, at 4299945344 00000100:00100000:0.0:1787697372.581570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697372.581601:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697373.605550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697373.605550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697374.629597:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697374.629598:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697375.141555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.141562:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697375.141566:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.141576:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5276 00000100:00080000:3.0:1787697375.141587:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697375.141592:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.141596:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141600:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697375.141605:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697375.141609:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.141611:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141613:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697375.141616:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141627:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697375.141648:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697375.141659:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697375.141675:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.141677:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697375.141679:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.141694:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5276 00000100:00080000:3.0:1787697375.141698:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697375.141701:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.141704:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141707:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697375.141710:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697375.141713:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.141716:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141719:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697375.141722:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697375.141731:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697375.141737:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697375.141749:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697375.141750:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697375.141799:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.141802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126205696:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697375.141820:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.141824:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697375.141826:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.141831:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5276 00000100:00080000:3.0:1787697375.141834:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697375.141837:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.141839:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141840:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697375.141842:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697375.141843:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697375.141853:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697375.141853:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141865:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697375.141878:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697375.141884:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787697375.141885:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697375.141892:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697375.141901:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697375.141908:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697375.141914:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.141916:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697375.141918:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697375.141920:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5276 00000100:00080000:3.0:1787697375.141923:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697375.141925:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.141927:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141928:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697375.141930:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697375.141935:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:2.0:1787697375.141940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697375.141946:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.141949:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697375.141953:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697375.141959:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697375.141963:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697375.141974:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697375.141979:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697375.141994:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697375.142002:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697375.142005:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697375.142007:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697375.142012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697375/real 1787697375] req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697375.142017:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5276 00000100:00080000:3.0:1787697375.142021:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697375.142023:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.142027:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.142034:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697375.142039:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697375.142047:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697375.142048:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697375.142050:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697375.142051:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697375.142055:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697375.142060:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697375.142068:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697375.142076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697375.142079:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697375.142107:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5286) 00000100:00000040:2.0:1787697375.142127:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697375.142137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697375.142142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697375.142147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126205696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697375.142159:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126205696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697375.142170:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697375.142190:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142200:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697375.142223:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142228:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697375.142238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697375.142242:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142249:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697375.142253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697375/real 1787697375] req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142271:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697375.142276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697375.142278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697375.142280:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142284:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697375.142288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697375.142301:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142305:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697375.142312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697375.142316:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142322:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697375.142325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697375/real 1787697375] req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697375.142359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697375.142362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697375.142365:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126205952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697375.142379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697375.142396:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142400:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697375.142408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697375.142412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142450:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697375.142455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697375/real 1787697375] req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142476:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697375.142481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697375.142483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697375.142485:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697375.142493:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697375.142508:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142515:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697375.142526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697375.142533:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697400 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142552:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697375.142557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697375/real 1787697375] req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697375.142575:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697375.142579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697375.142581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697375.142583:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697375.142588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697375.142591:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126205824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697375.142596:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697375.142600:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126205952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697375.142604:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697375.142608:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126206080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697375.142612:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697375.142614:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126206208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697400 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697375.142624:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697375.653532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697375.653561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697376.677563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697376.677563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697377.061543:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299950447, at 4299950464 00000100:00000040:2.0:1787697377.061551:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299950447, at 4299950464 00000100:00100000:2.0:1787697377.701589:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697378.725508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697378.725535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697378.725542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697379.749579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697379.749580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697380.261746:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.261764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126206336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697380.261796:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.261815:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697380.261850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697380.261873:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.261917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697380/real 1787697380] req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.261935:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.261950:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697380.261964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697380.261971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697380.261976:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.261989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126206336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697380.261998:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126206336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697380.262058:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697380.262078:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697380.262119:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262139:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697380.262155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697380.262166:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262181:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697380.262191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697380/real 1787697380] req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697380.262215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697380.262217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697380.262219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697380.262232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697380.262261:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262269:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697380.262281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697380.262288:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262297:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697380.262303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697380/real 1787697380] req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697380.262330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697380.262333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697380.262337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697380.262350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262364:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697380.262377:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262392:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697380.262404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697380.262411:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262446:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697380.262453:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697380/real 1787697380] req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262480:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697380.262486:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697380.262489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697380.262493:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697380.262505:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126206464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697380.262513:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697380.262527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126206592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697380.262535:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697380.262540:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126206720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697380.262555:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697380.262564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697380.262585:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262592:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697380.262604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697380.262611:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697405 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262620:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697380.262625:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697380/real 1787697380] req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262637:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697380.262645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697380.262653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697380.262656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697380.262667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697380.262675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126206848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697380.262680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126206848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697405 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697380.262688:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697380.773501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697380.773502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697381.797513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697381.797513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697382.181529:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299955567, at 4299955584 00000100:00000040:3.0:1787697382.181529:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299955567, at 4299955584 00000100:00100000:0.0:1787697382.821491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697382.821607:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697383.845531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697383.845538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697384.869554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697384.869728:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697385.381595:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697385.381608:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697385.381612:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697385.381622:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5286 00000100:00080000:0.0:1787697385.381636:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697385.381639:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697385.381642:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381645:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697385.381650:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697385.381652:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697385.381653:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381656:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697385.381659:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381674:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697385.381706:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697385.381718:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697385.381738:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697385.381741:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697385.381760:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697385.381765:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5286 00000100:00080000:0.0:1787697385.381771:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697385.381774:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697385.381778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381780:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697385.381785:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697385.381786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697385.381791:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697385.381799:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381804:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697385.381810:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697385.381811:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126206976:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787697385.381822:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697385.381833:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697385.381847:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697385.381851:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.381872:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697385.381874:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697385.381887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697385.381891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697385.381892:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697385.381896:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5286 00000100:00080000:0.0:1787697385.381901:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697385.381904:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697385.381907:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381910:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000800:00000100:2.0:1787697385.381912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381927:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697385.381942:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697385.381945:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.381947:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697385.381949:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697385.381954:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697385.381958:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697385.381962:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697385.381972:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697385.381983:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697385.381992:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697385.381997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697385/real 1787697385] req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697385.381998:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697385.382002:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697385.382013:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5286 00000100:00100000:2.0:1787697385.382018:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697385.382030:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697385.382037:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697385.382042:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697385.382043:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.382049:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697385.382053:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697385.382059:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697385.382062:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787697385.382063:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.382068:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697385.382071:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697385.382077:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697385.382080:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787697385.382083:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697385.382088:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697385.382097:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697385.382098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126206976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697385.382106:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126206976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697385.382122:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697385.382125:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787697385.382129:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697385.382133:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697385.382135:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697385.382138:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5286 00000100:00080000:0.0:1787697385.382141:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697385.382143:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697385.382146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697385.382148:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697385.382153:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697385.382156:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697385.382161:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697385.382165:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697385.382166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207104:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787697385.382168:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697385.382174:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697385.382181:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697385.382186:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697385.382190:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697385.382201:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697385.382204:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697385.382218:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697385.382225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697385.382238:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5296) 00000100:00000040:2.0:1787697385.382241:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382254:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697385.382261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697385/real 1787697385] req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382296:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697385.382304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697385.382307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697385.382311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697385.382325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697385.382345:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382354:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697385.382366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697385.382373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382382:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697385.382388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697385/real 1787697385] req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697385.382461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697385.382464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697385.382468:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697385.382482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697385.382505:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382513:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697385.382524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697385.382531:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382541:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697385.382546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697385/real 1787697385] req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382569:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382577:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697385.382585:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697385.382588:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697385.382593:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697385.382606:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126207104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697385.382615:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697385.382621:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126207232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697385.382628:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697385.382633:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126207360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697385.382638:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697385.382646:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382653:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697385.382672:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382679:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697385.382690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697385.382697:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697410 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382706:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697385.382712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697385/real 1787697385] req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697385.382733:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697385.382741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697385.382744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697385.382748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697385.382755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697385.382761:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126207488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697410 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697385.382769:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697385.893552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697385.893562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697386.917553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697386.917560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697387.301555:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299960687, at 4299960704 00000100:00000040:3.0:1787697387.301555:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299960687, at 4299960704 00000100:00100000:3.0:1787697387.941555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697387.941563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697388.965571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697388.965572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697389.989510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697389.989599:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697390.501897:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697390.501927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126207616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697390.501976:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697390.501999:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697390.503618:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697390.505921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697390.505966:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697390.505977:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697390.506016:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:2.0:1787697390.506029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697390.506042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697390.506056:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697390.506065:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697390.506074:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:2.0:1787697390.506600:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697390.506621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126208000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697390.506636:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697390.506644:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697390.506659:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:2.0:1787697390.506667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697390.506676:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126208128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697390.506688:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697390.506696:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697390.506707:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697391.013522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697391.013584:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697392.037540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697392.037741:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697392.421505:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299965809, at 4299965824 00000100:00000040:3.0:1787697392.421526:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299965809, at 4299965824 00000100:00100000:3.0:1787697393.061568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697393.061674:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000400:00000100:3.0:1787697393.573821:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787697393.573858:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787697393.573873:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697393.574636:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.576862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697390/real 1787697393] req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697415 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.576908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697393.576910:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697393.576929:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697393.576935:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697393.576969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:3.0:1787697393.576974:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697393.576989:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697393.577010:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697393.577018:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697393.577023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000800:00000100:3.0:1787697393.577029:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697393.577040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697393.577046:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697415 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126207616:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787697393.577078:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697390/real 1787697393] req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697393.577139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697393.577148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697393.577152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697393.577156:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207744:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787697393.577172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697390/real 1787697393] req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697393.577192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697393.577211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697393.577215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697393.577219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126207872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697393.577244:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126207616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697393.577260:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697393.577273:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126207744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697393.577280:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697393.577286:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126207872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697393.577294:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697393.577304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697390/real 1787697393] req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697393.577327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697393.577335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697393.577338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697393.577359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126208000:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787697393.577376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697390/real 1787697393] req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697393.577397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697393.577405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697393.577408:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697393.577412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697393.577417:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126208128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697393.577439:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126208000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697393.577448:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697393.577454:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126208128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697415 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697393.577461:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697394.085483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697394.085493:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697395.109542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697395.109542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697395.621555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697395.621577:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697395.621581:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697395.621591:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5296 00000100:00080000:2.0:1787697395.621602:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697395.621605:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621609:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621618:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697395.621623:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697395.621627:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621629:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621632:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697395.621635:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621645:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697395.621670:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126208256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697395.621686:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126208256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697395.621710:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697395.621714:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697395.621716:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697395.621720:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5296 00000100:00080000:2.0:1787697395.621725:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697395.621727:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621730:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621732:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697395.621735:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697395.621760:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621763:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621766:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697395.621771:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697395.621781:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697395.621788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697395.621804:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126208384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697395.621815:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126208384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697395.621826:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697395.621833:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697395.621837:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697395.621846:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5296 00000100:00080000:2.0:1787697395.621853:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697395.621856:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621859:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621861:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697395.621864:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697395.621867:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621870:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621872:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697395.621875:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697395.621880:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697395.621885:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697395.621897:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126208512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697395.621929:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126208512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697395.621940:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697395.621944:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697395.621946:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697395.621950:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5296 00000100:00080000:2.0:1787697395.621955:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697395.621961:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621967:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621972:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697395.621975:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697395.621978:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.621981:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.621984:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697395.621989:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697395.621996:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697395.622004:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697395.622015:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126208640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697395.622031:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126208640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697395.622043:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697395.622050:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697395.622053:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697395.622065:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5296 00000100:00080000:2.0:1787697395.622070:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697395.622077:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.622084:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.622088:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697395.622092:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697395.622098:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697395.622101:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697395.622103:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697395.622106:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697395.622111:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697395.622118:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697395.622132:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126208768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697395.622150:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126208768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697395.622164:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5306) 00000100:00100000:2.0:1787697396.133542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697396.136230:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697397.157578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697397.157584:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697397.541746:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299970927, at 4299970944 00000100:00000040:2.0:1787697397.541975:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299970927, at 4299970944 00000100:00100000:0.0:1787697398.181954:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697398.182033:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697399.205667:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697399.205813:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697399.205856:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697400.229535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697400.229637:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697400.741557:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697400.741566:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697400.741569:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697400.741580:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5301 00000100:00080000:3.0:1787697400.741590:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697400.741594:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.741599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.741602:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697400.741607:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697400.741611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.741613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.741616:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697400.741619:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697400.741633:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697400.741672:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126208896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697400.741687:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126208896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697400.741707:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697400.741721:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697400.741723:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697400.741726:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5301 00000100:00080000:3.0:1787697400.741729:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697400.741732:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.741735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.741737:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697400.741740:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697400.741742:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.741745:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.741747:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697400.741750:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697400.741831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697400.741836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697400.741844:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126209024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697400.741851:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126209024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697400.741868:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697400.741872:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697400.741874:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697400.741877:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5301 00000100:00080000:3.0:1787697400.741881:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697400.741884:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.741887:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.741895:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697400.741897:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697400.741900:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.741903:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.741905:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697400.741907:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697400.741912:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697400.741916:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697400.741926:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126209152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697400.742249:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126209152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697400.742255:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697400.742257:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697400.742259:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697400.742261:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5301 00000100:00080000:3.0:1787697400.742265:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697400.742266:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.742268:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.742269:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697400.742271:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697400.742273:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.742274:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.742276:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697400.742277:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697400.742281:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697400.742288:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697400.742295:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126209280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697400.742301:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126209280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697400.742306:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697400.742308:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697400.742309:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697400.742310:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5301 00000100:00080000:3.0:1787697400.742312:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697400.742314:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.742315:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.742316:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697400.742317:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697400.742319:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697400.742321:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697400.742322:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697400.742323:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697400.742326:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697400.742328:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697400.742332:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126209408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697400.742337:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126209408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697400.742351:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5311) 00000100:00100000:3.0:1787697401.253561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697401.253570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697402.277523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697402.277583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697402.661523:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299976049, at 4299976064 00000100:00000040:2.0:1787697402.661530:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299976049, at 4299976064 00000100:00100000:3.0:1787697403.301565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697403.301568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697404.325557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697404.325668:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697405.349523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697405.349523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697405.861815:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697405.861843:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697405.861847:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697405.861858:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5306 00000100:00080000:2.0:1787697405.861869:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697405.861873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.861877:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.861881:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697405.861885:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697405.861888:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.861891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.861894:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697405.861897:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697405.861909:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697405.861936:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126209536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697405.861951:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126209536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697405.861975:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697405.861979:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697405.861981:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697405.861985:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5306 00000100:00080000:2.0:1787697405.861989:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697405.861992:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.861995:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.861997:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697405.861999:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697405.862107:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.862111:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.862114:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697405.862116:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697405.862129:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697405.862136:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697405.862151:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126209664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697405.862164:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126209664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697405.862175:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697405.862180:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697405.862185:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697405.862193:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5306 00000100:00080000:2.0:1787697405.862200:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697405.862204:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.862210:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.862215:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697405.862219:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697405.862227:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.862233:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.862238:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697405.862245:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697405.862254:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697405.862262:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697405.862279:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126209792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697405.862314:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126209792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697405.862327:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697405.862334:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697405.862338:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697405.862345:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5306 00000100:00080000:2.0:1787697405.862352:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697405.862356:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.862358:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.862361:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697405.862365:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697405.862373:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.862381:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.862385:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697405.862391:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697405.862402:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697405.862409:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697405.862462:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126209920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697405.862481:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126209920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697405.862512:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697405.862519:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697405.862524:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697405.862538:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5306 00000100:00080000:2.0:1787697405.862545:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697405.862554:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.862559:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.862575:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697405.862578:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697405.862586:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697405.862593:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697405.862597:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697405.862604:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697405.862613:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697405.862622:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697405.862641:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126210048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697405.862679:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126210048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697405.862692:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5316) 00000100:00100000:3.0:1787697406.373562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697406.373645:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697407.397512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697407.397593:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697407.781504:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299981167, at 4299981184 00000100:00000040:3.0:1787697407.781506:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299981167, at 4299981184 00000100:00100000:3.0:1787697408.421553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697408.421556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697409.445539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697409.445541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697410.469543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697410.469637:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697410.981522:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697410.981534:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697410.981537:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697410.981547:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5311 00000100:00080000:0.0:1787697410.981558:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697410.981562:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981566:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981570:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697410.981575:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697410.981578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981581:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981584:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697410.981587:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981600:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697410.981625:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126210176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697410.981640:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126210176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697410.981662:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697410.981666:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697410.981668:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697410.981672:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5311 00000100:00080000:0.0:1787697410.981675:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697410.981677:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981680:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981694:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697410.981697:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697410.981701:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981704:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981706:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697410.981711:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697410.981720:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697410.981728:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697410.981742:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126210304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697410.981755:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126210304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697410.981765:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697410.981770:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697410.981773:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697410.981782:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5311 00000100:00080000:0.0:1787697410.981789:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697410.981796:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981799:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981802:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697410.981804:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697410.981807:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981812:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697410.981815:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697410.981821:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697410.981833:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697410.981847:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126210432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697410.981873:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126210432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697410.981882:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697410.981886:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697410.981888:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697410.981895:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5311 00000100:00080000:0.0:1787697410.981901:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697410.981906:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981913:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697410.981917:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697410.981923:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.981930:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.981934:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697410.981938:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697410.981943:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697410.981950:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697410.981961:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126210560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697410.981981:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126210560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697410.981998:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697410.982007:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697410.982011:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697410.982019:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5311 00000100:00080000:0.0:1787697410.982025:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697410.982031:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.982034:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.982037:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697410.982039:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697410.982046:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697410.982053:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697410.982058:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697410.982065:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697410.982075:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697410.982081:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697410.982095:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126210688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697410.982110:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126210688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697410.982125:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5322) 00000100:00100000:2.0:1787697411.493528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697411.493528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697412.517547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697412.517651:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697412.901547:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299986287, at 4299986304 00000100:00000040:3.0:1787697412.901547:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299986287, at 4299986304 00000100:00100000:2.0:1787697413.541556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697413.541556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697413.541592:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697414.565565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697414.565576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697414.565580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697414.693678:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697415.589528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697415.589539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697415.589624:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697415.717681:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697416.101702:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.101716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126210816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697416.101739:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.101748:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697416.101778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697416.101792:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.101819:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697416] req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.101833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.101843:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697416.101852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697416.101855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697416.101859:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.101867:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126210816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697416.101890:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126210816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697416.101899:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697416.101912:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.101921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126210944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697416.101934:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.101941:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697416.101953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697416.101960:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.101967:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697416.101971:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697416] req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.101977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.101983:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697416.101987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697416.101989:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697416.101991:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102003:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126210944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697416.102006:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697416.102019:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102024:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697416.102030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697416.102034:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102039:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697416.102042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697416] req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697416.102065:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697416.102067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697416.102071:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697416.102089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697416.102108:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102117:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697416.102125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697416.102129:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102135:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697416.102138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697416] req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102149:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697416.102154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697416.102155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697416.102157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697416.102165:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126210944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697416.102174:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697416.102177:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126211072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697416.102182:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697416.102184:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126211200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697416.102188:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697416.102193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697416.102204:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102208:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697416.102215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697416.102218:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697441 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697416.102226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697416/real 1787697416] req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697416.102237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697416.102253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697416.102256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697416.102260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697416.102268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697416.102274:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126211328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697441 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697416.102281:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697416.613534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697416.613549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697416.613565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697416.741649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697417.637573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697417.637577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697417.637654:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697417.765560:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697418.021551:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299991407, at 4299991424 00000100:00000040:2.0:1787697418.021614:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299991407, at 4299991424 00000100:00100000:0.0:1787697418.661564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697418.661567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697418.662575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697418.789593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697419.685483:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697419.685485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697419.685490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697419.813690:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697420.709585:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697420.711413:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697420.837580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697421.221584:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697421.221593:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697421.221614:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697421.221625:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5322 00000100:00080000:0.0:1787697421.221634:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697421.221638:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221642:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221644:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697421.221648:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697421.221650:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221652:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221654:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697421.221656:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221664:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697421.221682:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697421.221693:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697421.221710:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697421.221712:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697421.221713:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697421.221716:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5322 00000100:00080000:0.0:1787697421.221719:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697421.221720:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221721:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221723:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697421.221725:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697421.221727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221730:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221736:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697421.221742:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697421.221744:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.221755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126211456:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787697421.221756:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697421.221763:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697421.221776:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697421.221778:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.221803:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221813:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697421.221824:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697421.221827:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697421.221829:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697421.221833:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5322 00000100:00080000:0.0:1787697421.221837:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787697421.221842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221844:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221852:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697421.221854:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697421.221856:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221861:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787697421.221863:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697421.221864:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697421.221871:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697421.221877:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697421.221885:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697421.221901:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.221905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697421/real 1787697421] req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697421.221909:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697421.221916:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697421.221925:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697421.221929:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5322 00000100:00100000:2.0:1787697421.221933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697421.221935:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697421.221941:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221946:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221949:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697421.221950:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697421.221953:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697421.221954:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.221960:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.221963:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697421.221969:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787697421.221974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787697421.221976:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697421.221981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787697421.221983:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697421.221988:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697421.221997:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697421.221999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126211456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697421.222010:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126211456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697421.222020:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697421.222021:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787697421.222033:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697421.222037:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:2.0:1787697421.222038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697421.222041:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697421.222049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211584:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787697421.222054:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5322 00000100:00080000:0.0:1787697421.222060:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697421.222063:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697421.222067:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.222075:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697421.222076:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697421.222080:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697421.222083:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:2.0:1787697421.222088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697421.222089:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697421.222091:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697421.222092:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697421.222096:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697421.222101:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697421.222103:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697421.222106:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697421.222115:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787697421.222119:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697421.222123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697421/real 1787697421] req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697421.222149:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697421.222152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697421.222158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697421.222164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697421.222173:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5332) 00000100:00000040:2.0:1787697421.222176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697421.222191:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222196:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697421.222202:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697421.222206:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222221:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697421.222224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697421/real 1787697421] req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697421.222240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697421.222242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697421.222244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697421.222252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697421.222263:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222267:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697421.222273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697421.222277:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222285:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697421.222293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697421/real 1787697421] req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697421.222332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697421.222335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697421.222339:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697421.222356:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126211584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697421.222364:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697421.222370:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126211712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697421.222377:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697421.222381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126211840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697421.222388:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697421.222397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697421.222415:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222436:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697421.222449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697421.222455:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697446 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222464:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697421.222469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697421/real 1787697421] req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697421.222488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697421.222496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697421.222499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697421.222502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697421.222510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126211968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697421.222515:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126211968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697446 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697421.222522:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697421.733499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697421.733555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697421.861637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697422.757662:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697422.758336:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697422.885611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697423.141513:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4299996527, at 4299996544 00000100:00000040:3.0:1787697423.141513:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4299996527, at 4299996544 00000100:00100000:0.0:1787697423.782317:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697423.784295:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697423.909600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697424.805595:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697424.805696:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697424.933640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697425.829538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697425.829538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697425.957588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697426.341987:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.341999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126212096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697426.342026:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342034:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697426.342067:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697426.342082:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697426/real 1787697426] req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697426.342141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697426.342145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697426.342149:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126212096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697426.342182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697426.342204:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342228:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697426.342240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697426.342246:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342257:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697426.342262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697426/real 1787697426] req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697426.342288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697426.342292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697426.342295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697426.342315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697426.342336:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342343:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697426.342354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697426.342361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342369:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697426.342374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697426/real 1787697426] req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342394:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697426.342401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697426.342405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697426.342408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697426.342443:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697426.342468:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342476:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697426.342486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697426.342492:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342500:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697426.342505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697426/real 1787697426] req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342524:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697426.342531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697426.342534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697426.342536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697426.342544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697426.342562:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342566:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697426.342573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697426.342577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697451 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342582:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697426.342585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697426/real 1787697426] req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342592:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697426.342597:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697426.342601:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697426.342602:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697426.342604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697426.342609:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697426.342615:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126212096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697426.342623:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697426.342630:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126212224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697426.342648:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697426.342654:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126212352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697426.342661:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697426.342666:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126212480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697426.342672:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697426.342676:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126212608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697451 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697426.342682:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697426.853517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697426.853623:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697426.981572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697427.877535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697427.877563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697428.005591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697428.261639:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300001647, at 4300001664 00000100:00000040:2.0:1787697428.264263:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300001647, at 4300001666 00000100:00100000:3.0:1787697428.901504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697428.901599:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697429.029599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697429.925519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697429.925519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697429.925741:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697430.053549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697430.949539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697430.949539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697430.949539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697431.077617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697431.461677:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.461701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126212736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697431.461732:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.461767:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697431.461796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697431.461813:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.461846:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697431/real 1787697431] req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.461881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.461899:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697431.461908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697431.461913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697431.461919:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.461933:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126212736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697431.461947:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126212736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697431.461975:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697431.461993:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697431.462034:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462071:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697431.462092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697431.462104:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462134:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697431.462145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697431/real 1787697431] req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697431.462184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697431.462187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697431.462190:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462198:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697431.462203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697431.462222:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462230:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697431.462248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697431.462255:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462264:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697431.462269:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697431/real 1787697431] req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697431.462295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697431.462298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697431.462302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126212992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697431.462314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126212864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697431.462321:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697431.462327:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126212992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697431.462333:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697431.462341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126213120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697431.462372:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462379:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697431.462391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697431.462398:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462407:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697431.462412:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697431/real 1787697431] req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462444:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697431.462460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697431.462463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697431.462467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126213120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697431.462480:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462488:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126213248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697431.462504:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462511:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697431.462521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697431.462528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697456 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462536:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697431.462541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697431/real 1787697431] req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697431.462560:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697431.462566:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697431.462568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697431.462570:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697431.462574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126213248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697431.462577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126213120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697431.462581:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697431.462585:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126213248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697456 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697431.462598:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697431.973529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697431.973530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697431.973543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697432.101547:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697432.997553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697432.997563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697432.997651:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697433.125693:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697433.381504:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300006767, at 4300006784 00000100:00000040:3.0:1787697433.381504:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300006767, at 4300006784 00000100:00100000:0.0:1787697434.021538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697434.021538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697434.021539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697434.149550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697435.045487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697435.045551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697435.173605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697436.069523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697436.069524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697436.197573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697436.581567:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697436.581575:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697436.581578:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697436.581586:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5337 00000100:00080000:2.0:1787697436.581595:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697436.581597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581602:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697436.581606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697436.581608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581611:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697436.581614:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581626:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697436.581650:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126213376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697436.581681:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126213376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697436.581697:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697436.581701:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697436.581702:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697436.581706:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5337 00000100:00080000:2.0:1787697436.581711:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697436.581714:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581717:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581719:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697436.581722:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697436.581729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581743:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697436.581746:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697436.581755:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697436.581761:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697436.581770:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126213504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697436.581781:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126213504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697436.581801:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697436.581805:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697436.581806:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697436.581814:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5337 00000100:00080000:2.0:1787697436.581818:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697436.581820:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581822:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581824:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697436.581826:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697436.581828:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581833:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581836:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697436.581838:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697436.581842:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697436.581845:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697436.581855:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126213632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697436.581865:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126213632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697436.581873:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697436.581880:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697436.581885:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697436.581893:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5337 00000100:00080000:2.0:1787697436.581901:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697436.581906:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581913:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581917:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697436.581920:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697436.581928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.581940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.581944:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697436.581948:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697436.581954:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697436.581962:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697436.581972:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126213760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697436.581987:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126213760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697436.582014:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697436.582024:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697436.582027:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697436.582045:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5337 00000100:00080000:2.0:1787697436.582051:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697436.582054:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.582057:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.582060:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697436.582063:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697436.582067:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697436.582070:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697436.582074:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697436.582078:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697436.582085:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697436.582090:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697436.582100:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126213888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697436.582143:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126213888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697436.582171:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5347) 00000100:00100000:2.0:1787697437.093524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697437.093524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697437.221602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697438.117542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697438.117640:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697438.245593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697438.501544:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300011887, at 4300011904 00000100:00000040:2.0:1787697438.501646:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300011887, at 4300011904 00000100:00100000:2.0:1787697439.141536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697439.141536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697439.269605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697440.165687:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697440.167035:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697440.293578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697441.189536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697441.189549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697441.317607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697441.701536:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697441.701546:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697441.701551:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697441.701562:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5342 00000100:00080000:2.0:1787697441.701572:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697441.701576:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701580:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701584:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697441.701588:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697441.701592:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701596:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701599:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697441.701603:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701633:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697441.701658:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126214016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697441.701674:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126214016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697441.701695:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697441.701698:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697441.701700:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697441.701703:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5342 00000100:00080000:2.0:1787697441.701706:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697441.701708:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701711:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697441.701713:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697441.701715:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701717:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701719:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697441.701721:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697441.701729:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697441.701733:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697441.701745:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126214144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697441.701764:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126214144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697441.701772:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697441.701780:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697441.701781:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697441.701784:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5342 00000100:00080000:2.0:1787697441.701787:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697441.701790:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701793:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701796:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697441.701798:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697441.701804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701816:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697441.701822:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697441.701831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697441.701837:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697441.701848:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126214272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697441.701858:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126214272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697441.701872:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697441.701876:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697441.701878:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697441.701883:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5342 00000100:00080000:2.0:1787697441.701887:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697441.701891:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701896:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701905:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697441.701907:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697441.701909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701912:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697441.701914:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697441.701919:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697441.701922:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697441.701937:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126214400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697441.701955:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126214400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697441.701963:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697441.701970:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697441.701974:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697441.701980:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5342 00000100:00080000:2.0:1787697441.701983:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697441.701989:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.701995:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.701998:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697441.702001:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697441.702005:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697441.702014:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697441.702019:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697441.702026:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697441.702034:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697441.702046:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697441.702059:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126214528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697441.702066:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126214528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697441.702076:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5352) 00000100:00100000:3.0:1787697442.213518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697442.213838:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697442.341545:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697443.237486:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697443.237502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697443.365614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697443.621534:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300017007, at 4300017024 00000100:00000040:3.0:1787697443.621534:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300017007, at 4300017024 00000100:00100000:0.0:1787697444.261524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697444.261524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697444.389598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697445.285544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697445.285549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697445.413653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697446.309530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697446.309620:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697446.437657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697446.821544:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697446.821555:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697446.821561:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697446.821573:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5347 00000100:00080000:2.0:1787697446.821587:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697446.821591:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821596:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821600:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697446.821606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697446.821609:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821611:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821614:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697446.821617:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821632:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697446.821660:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126214656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697446.821711:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126214656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697446.821741:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697446.821744:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697446.821745:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697446.821749:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5347 00000100:00080000:2.0:1787697446.821752:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697446.821754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821758:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697446.821760:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697446.821762:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821764:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821765:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697446.821767:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697446.821775:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697446.821782:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697446.821790:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126214784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697446.821799:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126214784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697446.821807:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697446.821811:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697446.821812:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697446.821823:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5347 00000100:00080000:2.0:1787697446.821827:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697446.821830:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821833:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821835:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697446.821837:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697446.821840:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821843:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821845:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697446.821847:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697446.821863:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697446.821868:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697446.821877:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126214912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697446.821883:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126214912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697446.821889:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697446.821891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697446.821892:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697446.821895:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5347 00000100:00080000:2.0:1787697446.821898:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697446.821899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821901:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821902:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697446.821904:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697446.821905:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821913:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697446.821914:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697446.821918:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697446.821922:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697446.821927:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126215040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697446.821933:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126215040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697446.821938:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697446.821939:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697446.821940:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697446.821942:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5347 00000100:00080000:2.0:1787697446.821944:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697446.821946:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821947:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821948:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697446.821949:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697446.821951:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697446.821954:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697446.821955:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697446.821958:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697446.821961:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697446.821965:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697446.821973:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126215168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697446.821987:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126215168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697446.821999:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5357) 00000100:00100000:3.0:1787697447.333566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697447.333694:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697447.461607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697448.357566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697448.357593:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697448.485601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697448.741497:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300022127, at 4300022144 00000100:00000040:2.0:1787697448.741498:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300022127, at 4300022144 00000100:00100000:3.0:1787697449.381512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697449.381638:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697449.509619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697450.405520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697450.405528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697450.533610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697451.429561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697451.429711:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697451.557645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697451.941555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697451.941568:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697451.941574:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697451.941586:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5352 00000100:00080000:2.0:1787697451.941599:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697451.941602:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.941607:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.941611:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697451.941616:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697451.941620:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.941623:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.941626:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697451.941630:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697451.941649:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697451.941680:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126215296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697451.941714:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126215296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697451.941748:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697451.941752:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697451.941754:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697451.941758:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5352 00000100:00080000:2.0:1787697451.941763:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697451.941779:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.941783:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.941786:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697451.941789:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697451.941793:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.941796:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.941798:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697451.941801:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697451.941812:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697451.941829:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697451.941844:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126215424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697451.941855:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126215424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697451.941865:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697451.941869:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697451.941872:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697451.941895:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5352 00000100:00080000:2.0:1787697451.941901:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697451.941905:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.941908:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.941911:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697451.941914:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697451.941917:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.941920:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.941922:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697451.941926:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697451.941934:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697451.941940:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697451.941954:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126215552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697451.941968:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126215552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697451.941978:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697451.941982:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697451.941984:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697451.941991:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5352 00000100:00080000:2.0:1787697451.941997:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697451.942002:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.942009:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.942012:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697451.942014:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697451.942018:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.942032:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.942038:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697451.942044:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697451.942053:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697451.942060:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697451.942074:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126215680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697451.942107:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126215680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697451.942120:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697451.942126:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697451.942129:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697451.942147:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5352 00000100:00080000:2.0:1787697451.942153:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697451.942158:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.942162:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.942164:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697451.942167:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697451.942171:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697451.942177:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697451.942182:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697451.942186:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697451.942191:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697451.942198:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697451.942212:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126215808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697451.942233:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126215808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697451.942249:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5363) 00000100:00100000:2.0:1787697452.453526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697452.453526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697452.581605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697453.477577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697453.478365:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697453.607090:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697453.861504:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300027247, at 4300027264 00000100:00000040:3.0:1787697453.861504:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300027247, at 4300027264 00000100:00100000:2.0:1787697454.501521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697454.501521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697454.629610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697455.525531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697455.525581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697455.653667:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697456.549586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697456.549590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697456.549594:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697456.677670:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697457.061597:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697457.061607:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697457.061611:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697457.061622:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5358 00000100:00080000:0.0:1787697457.061634:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697457.061636:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.061640:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.061643:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697457.061648:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697457.061650:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.061652:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.061656:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697457.061659:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697457.061700:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697457.061735:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126215936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697457.061752:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126215936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.061778:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697457.061782:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697457.061801:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697457.061806:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5358 00000100:00080000:0.0:1787697457.061810:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697457.061812:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.061815:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.061816:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697457.061819:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697457.061825:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.061831:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.061835:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697457.061840:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697457.061852:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697457.061857:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697457.061870:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126216064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697457.061882:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126216064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.061895:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697457.061904:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697457.061908:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697457.061916:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5358 00000100:00080000:0.0:1787697457.061920:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697457.061924:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.061927:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.061929:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697457.061931:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697457.061941:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.061943:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.061946:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697457.061948:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697457.061954:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697457.061959:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697457.061971:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126216192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697457.061981:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126216192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.062000:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697457.062010:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697457.062013:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697457.062019:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5358 00000100:00080000:0.0:1787697457.062025:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697457.062033:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.062037:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.062040:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697457.062043:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697457.062047:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.062052:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.062056:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697457.062063:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697457.062071:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697457.062080:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697457.062094:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126216320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697457.062119:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126216320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697457.062135:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697457.062138:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697457.062141:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697457.062144:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5358 00000100:00080000:0.0:1787697457.062149:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697457.062153:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.062156:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.062159:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697457.062162:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697457.062169:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697457.062177:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697457.062182:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697457.062188:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697457.062195:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697457.062201:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697457.062215:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126216448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697457.062238:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126216448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697457.062251:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5368) 00000100:00100000:2.0:1787697457.573551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697457.573551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697457.573564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697457.701617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697458.597539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697458.597539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697458.597539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697458.725696:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697458.981541:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300032367, at 4300032384 00000100:00000040:3.0:1787697458.981541:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300032367, at 4300032384 00000100:00100000:3.0:1787697459.621544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697459.621566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697459.621702:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697459.749611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697460.133817:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:2.0:1787697460.133891:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697460.133913:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697460.133962:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126215936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697482 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697460.133982:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697460.133992:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126216064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697482 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697460.134024:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697460.134037:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126216192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697482 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697460.134051:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697460.134064:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126216320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697482 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697460.134079:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697460.134091:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126216448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697482 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697460.645501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697460.645501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697460.645540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697460.773660:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697461.669522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697461.669524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697461.669555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697461.797626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697462.181846:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.181874:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126216576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697462.181913:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.181940:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697462.181981:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697462.182003:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697462/real 1787697462] req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182097:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697462.182150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697462.182155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697462.182164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126216576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697462.182200:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126216576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697462.182213:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697462.182231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126216704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697462.182254:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182262:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697462.182274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697462.182282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182292:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697462.182298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697462/real 1787697462] req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182333:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697462.182340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697462.182345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697462.182349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126216704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697462.182362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126216832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697462.182382:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182388:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697462.182400:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697462.182406:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182414:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697462.182449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697462/real 1787697462] req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697462.182488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697462.182491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697462.182494:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126216832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697462.182505:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126216960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697462.182524:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182530:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697462.182539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697462.182546:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182554:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697462.182559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697462/real 1787697462] req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182587:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697462.182607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697462.182609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697462.182613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126216960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697462.182625:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182633:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697462.182644:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182651:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697462.182662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697462.182668:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697487 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182675:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697462.182680:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697462/real 1787697462] req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697462.182703:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697462.182711:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697462.182715:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697462.182718:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697462.182724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697462.182729:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126216704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697462.182736:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697462.182742:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126216832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697462.182747:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697462.182752:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126216960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697462.182757:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697462.182761:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126217088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697487 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697462.182767:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697462.693514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697462.693514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697462.693514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697462.821608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697463.717493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697463.717538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697463.845645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697464.102204:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300037487, at 4300037504 00000100:00000040:3.0:1787697464.102437:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300037487, at 4300037505 00000100:00100000:2.0:1787697464.741573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697464.741607:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697464.869637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697465.765550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697465.765550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697465.893660:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697466.789545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697466.789548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697466.917585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697467.301741:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.301765:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126217216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697467.301810:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.301831:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697467.301874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697467.301897:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.301948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697467/real 1787697467] req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.301970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.301990:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697467.302008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697467.302016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697467.302026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126217216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697467.302069:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126217216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697467.302085:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697467.302102:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697467.302121:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302133:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697467.302147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697467.302154:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302174:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697467.302180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697467/real 1787697467] req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697467.302199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697467.302200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697467.302209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697467.302217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697467.302231:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302236:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697467.302243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697467.302247:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302253:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697467.302256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697467/real 1787697467] req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697467.302272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697467.302273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697467.302276:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697467.302290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697467.302303:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302307:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697467.302313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697467.302317:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302322:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697467.302325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697467/real 1787697467] req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697467.302340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697467.302342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697467.302343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697467.302354:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697467.302366:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302370:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697467.302376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697467.302379:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697492 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302384:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697467.302388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697467/real 1787697467] req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697467.302398:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697467.302402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697467.302403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697467.302405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697467.302409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697467.302412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126217344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697467.302444:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697467.302448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126217472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697467.302452:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697467.302455:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126217600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697467.302459:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697467.302461:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126217728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697492 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697467.302465:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697467.813516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697467.813516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697467.941626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697468.837540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697468.837547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697468.965590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697469.221534:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300042607, at 4300042624 00000100:00000040:3.0:1787697469.221534:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300042607, at 4300042624 00000100:00100000:2.0:1787697469.861572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697469.861573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697469.989825:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697470.885583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697470.885610:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697471.013647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697471.909516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697471.910496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697472.037608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697472.421720:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.421740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126217856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697472.421782:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.421802:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697472.421860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697472.421880:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.421917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697472/real 1787697472] req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.421936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.421960:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697472.421972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697472.421978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697472.421986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422016:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126217856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697472.422028:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126217856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697472.422042:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697472.422061:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697472.422095:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422107:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697472.422121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697472.422135:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422143:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697472.422147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697472/real 1787697472] req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422159:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697472.422163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697472.422165:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697472.422167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126217984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697472.422175:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697472.422188:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422193:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697472.422199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697472.422203:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422217:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697472.422221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697472/real 1787697472] req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697472.422237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697472.422239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697472.422241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697472.422249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697472.422261:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422266:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697472.422272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697472.422275:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422281:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697472.422288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697472/real 1787697472] req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422295:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697472.422305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697472.422306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697472.422308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697472.422315:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126217984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697472.422319:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697472.422323:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126218112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697472.422326:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697472.422329:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126218240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697472.422333:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697472.422338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697472.422354:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422358:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697472.422365:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697472.422368:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697497 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422374:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697472.422377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697472/real 1787697472] req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697472.422398:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697472.422402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697472.422404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697472.422405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697472.422409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697472.422448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126218368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697497 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697472.422457:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697472.933544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697472.933544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697473.061589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697473.957521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697473.957521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697474.085658:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697474.341559:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300047727, at 4300047744 00000100:00000040:2.0:1787697474.341560:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300047727, at 4300047744 00000100:00100000:0.0:1787697474.981549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697474.981549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697475.109618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697476.005540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697476.005544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697476.133608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697477.029538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697477.029578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697477.157566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697477.541548:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697477.541556:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697477.541561:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697477.541569:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5378 00000100:00080000:3.0:1787697477.541581:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697477.541598:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541602:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541605:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697477.541610:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697477.541613:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541615:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541618:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697477.541621:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541634:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697477.541660:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697477.541672:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697477.541694:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697477.541697:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697477.541698:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697477.541702:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5378 00000100:00080000:3.0:1787697477.541706:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697477.541708:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541710:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541712:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697477.541714:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697477.541717:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541719:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541721:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697477.541724:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697477.541733:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697477.541747:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697477.541747:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697477.541761:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697477.541777:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.541780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126218496:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697477.541797:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697477.541800:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697477.541802:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697477.541807:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5378 00000100:00080000:3.0:1787697477.541811:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697477.541816:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697477.541817:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697477.541821:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541825:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697477.541829:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697477.541835:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697477.541839:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541842:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541845:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697477.541846:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697477.541850:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697477.541859:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697477.541873:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697477.541883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541888:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697477.541898:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697477.541904:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697477.541909:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697477.541914:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697477.541922:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5378 00000100:00080000:3.0:1787697477.541926:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697477.541928:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541930:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541931:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697477.541933:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697477.541935:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697477.541937:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.541939:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697477.541941:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697477.541945:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697477.541948:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697477/real 1787697477] req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697477.541950:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697477.541960:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697477.541967:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.541973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697477.541974:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697477.541986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697477.541988:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697477.541991:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697477.542000:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5378 00000100:00080000:2.0:1787697477.542003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697477.542006:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697477.542008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787697477.542014:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697477.542018:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697477.542020:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697477.542029:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126218496:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787697477.542036:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697477.542039:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697477.542046:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697477.542048:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697477.542050:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787697477.542054:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126218496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697477.542054:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697477.542062:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697477.542068:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:2.0:1787697477.542072:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697477.542077:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697477.542084:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542091:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697477.542091:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5388) 00000100:00100000:2.0:1787697477.542104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697477.542120:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542127:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697477.542139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697477.542146:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542156:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697477.542161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697477/real 1787697477] req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542171:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697477.542188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697477.542191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697477.542194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697477.542215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697477.542229:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542234:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697477.542241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697477.542246:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542251:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697477.542255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697477/real 1787697477] req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542266:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697477.542272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697477.542275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697477.542279:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697477.542296:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697477.542319:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542338:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697477.542348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697477.542354:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542360:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697477.542363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697477/real 1787697477] req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542376:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697477.542380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697477.542382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697477.542384:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126218880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697477.542391:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126218624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697477.542400:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697477.542404:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126218752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697477.542409:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697477.542412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126218880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697477.542416:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697477.542439:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697477.542452:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542457:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697477.542465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697477.542470:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697502 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542475:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697477.542478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697477/real 1787697477] req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697477.542490:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697477.542503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697477.542505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697477.542508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697477.542512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697477.542515:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126219008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697502 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697477.542519:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697478.053515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697478.053515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697478.053544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697478.181573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697479.077510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697479.077510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697479.077538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697479.205527:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697479.461513:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300052847, at 4300052864 00000100:00000040:3.0:1787697479.461513:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300052847, at 4300052864 00000100:00100000:3.0:1787697480.101519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697480.101520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697480.101524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697480.229589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697481.125500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697481.125502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697481.125503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697481.253561:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697482.149524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697482.149524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697482.149524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697482.277591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697482.661478:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697482.661486:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697482.661489:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697482.661497:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5383 00000100:00080000:3.0:1787697482.661505:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697482.661507:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697482.661511:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661513:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697482.661537:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697482.661540:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697482.661543:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661545:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697482.661547:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661557:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697482.661574:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697482.661584:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697482.661601:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697482.661603:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697482.661605:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697482.661607:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5383 00000100:00080000:3.0:1787697482.661610:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697482.661611:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697482.661613:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661614:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697482.661616:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697482.661618:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697482.661619:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661620:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697482.661622:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697482.661629:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697482.661632:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697482.661638:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697482.661645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697482.661654:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.661664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126219136:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697482.661664:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697482.661669:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697482.661672:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697482.661675:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5383 00000100:00080000:3.0:1787697482.661677:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697482.661679:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697482.661680:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661682:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697482.661683:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697482.661685:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697482.661686:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661688:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787697482.661692:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697482.661692:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697482.661696:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697482.661703:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697482.661707:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697482.661714:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697482.661721:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697482.661726:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697482.661728:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697482.661734:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787697482.661741:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697482.661745:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5383 00000100:00080000:3.0:1787697482.661751:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697482.661755:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697482.661758:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697482.661761:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661766:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697482.661770:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697482.661777:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697482.661781:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661784:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697482.661786:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697482.661792:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697482.661797:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697482.661801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697482/real 1787697482] req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697482.661807:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697482.661817:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.661821:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697482.661826:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697482.661832:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697482.661835:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697482.661839:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697482.661840:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5383 00000100:00080000:3.0:1787697482.661846:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697482.661851:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697482.661853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787697482.661855:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697482.661859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:3.0:1787697482.661859:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697482.661863:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697482.661865:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697482.661873:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697482.661876:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126219136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697482.661879:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697482.661883:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697482.661885:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697482.661887:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126219136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697482.661890:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697482.661895:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:2.0:1787697482.661899:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697482.661904:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697482.661910:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.661918:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697482.661919:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5393) 00000100:00100000:2.0:1787697482.661933:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697482.661948:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.661954:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697482.661963:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697482.661973:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.661980:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697482.661984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697482/real 1787697482] req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.661991:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.661997:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697482.662001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697482.662003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697482.662005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697482.662013:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697482.662025:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662030:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697482.662035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697482.662039:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662052:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697482.662055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697482/real 1787697482] req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697482.662072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697482.662074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697482.662075:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697482.662082:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697482.662094:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662098:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697482.662104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697482.662108:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662113:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697482.662121:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697482/real 1787697482] req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662128:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697482.662137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697482.662139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697482.662141:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697482.662148:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126219264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697482.662152:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697482.662155:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126219392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697482.662159:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697482.662161:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126219520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697482.662165:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697482.662170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697482.662185:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662190:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697482.662195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697482.662199:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697507 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662205:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697482.662208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697482/real 1787697482] req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697482.662220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697482.662224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697482.662225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697482.662227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697482.662231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697482.662399:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126219648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697507 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697482.662405:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697483.173509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697483.173509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697483.173522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697483.301598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697484.197536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697484.197536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697484.197637:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697484.325609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697484.581530:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300057967, at 4300057984 00000100:00000040:3.0:1787697484.581530:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300057967, at 4300057984 00000100:00100000:0.0:1787697485.221503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697485.221503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697485.349595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697486.245537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697486.245537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697486.373556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697487.269487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697487.269492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697487.397548:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697487.781762:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.781783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126219776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697487.781814:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.781837:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697487.781885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697487.781907:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.781954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697487/real 1787697487] req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.781981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782015:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697487.782041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697487.782050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697487.782060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126219776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697487.782119:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126219776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697487.782136:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697487.782169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697487.782197:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782204:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697487.782217:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697487.782225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782234:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697487.782241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697487/real 1787697487] req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697487.782269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697487.782272:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697487.782284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126219904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697487.782298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697487.782320:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782328:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697487.782340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697487.782347:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782357:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697487.782362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697487/real 1787697487] req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697487.782390:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697487.782393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697487.782396:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697487.782444:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697487.782466:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782474:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697487.782485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697487.782498:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782505:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697487.782511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697487/real 1787697487] req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782531:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697487.782538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697487.782541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697487.782545:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782552:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697487.782575:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782584:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697487.782597:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782605:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697487.782617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697487.782624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697512 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782633:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697487.782639:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697487/real 1787697487] req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782659:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697487.782668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697487.782676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697487.782679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697487.782683:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697487.782691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697487.782697:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126219904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697487.782719:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697487.782725:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126220032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697487.782732:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697487.782737:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126220160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697487.782744:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697487.782748:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126220288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697512 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697487.782755:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697488.293567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697488.293567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697488.421586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697489.317558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697489.317677:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697489.445669:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697489.701571:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300063087, at 4300063104 00000100:00000040:2.0:1787697489.701696:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300063087, at 4300063104 00000100:00100000:0.0:1787697490.341569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697490.341581:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697490.469652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697491.365546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697491.365546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697491.493634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697492.389536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697492.389558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697492.517646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697492.901765:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.901786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126220416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697492.901826:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.901848:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697492.901897:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697492.901940:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.901986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697492/real 1787697492] req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902015:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697492.902057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697492.902074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697492.902079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126220416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697492.902104:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126220416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697492.902126:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697492.902150:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697492.902177:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902183:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697492.902192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697492.902197:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902210:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697492.902215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697492/real 1787697492] req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697492.902232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697492.902234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697492.902236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697492.902244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697492.902257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902262:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697492.902268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697492.902272:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902278:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697492.902286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697492/real 1787697492] req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902299:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697492.902303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697492.902305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697492.902307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697492.902314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697492.902333:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902339:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697492.902350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697492.902357:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902366:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697492.902371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697492/real 1787697492] req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902394:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697492.902399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697492.902401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697492.902403:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697492.902411:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697492.902447:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902452:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697492.902460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697492.902464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697517 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902470:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697492.902473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697492/real 1787697492] req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697492.902489:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697492.902494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697492.902496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697492.902499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697492.902506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126220928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697492.902512:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126220544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697492.902520:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697492.902526:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126220672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697492.902533:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697492.902539:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126220800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697492.902546:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697492.902550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126220928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697517 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697492.902557:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697493.413537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697493.413537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697493.541691:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697494.437554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697494.437554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697494.565637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697494.821501:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300068207, at 4300068224 00000100:00000040:3.0:1787697494.821501:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300068207, at 4300068224 00000100:00100000:0.0:1787697495.461522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697495.461525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697495.589624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697496.485545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697496.485545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697496.613604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697497.509488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697497.509541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697497.637637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697498.021942:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.021951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126221056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697498.021995:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022001:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697498.022035:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697498.022047:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022077:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022083:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697498.022088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697498.022091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697498.022093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126221056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697498.022101:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697498.022131:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022135:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697498.022143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697498.022147:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022154:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697498.022157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697498.022173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697498.022174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697498.022177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697498.022184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697498.022200:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022204:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697498.022211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697498.022215:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022220:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697498.022223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697498.022238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697498.022240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697498.022242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697498.022249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697498.022260:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022268:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697498.022276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697498.022280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022286:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697498.022292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697498.022319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697498.022322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697498.022326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697498.022339:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697498.022360:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022367:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697498.022382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697498.022387:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697523 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022393:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697498.022396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697498/real 1787697498] req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697498.022408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697498.022413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697498.022415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697498.022437:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697498.022446:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697498.022452:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126221056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697498.022462:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697498.022470:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126221184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697498.022477:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697498.022483:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126221312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697498.022490:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697498.022503:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126221440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697498.022510:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697498.022515:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126221568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697523 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697498.022522:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697498.533541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697498.533546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697498.661581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697499.557485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697499.557491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697499.557571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697499.685596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697499.941528:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300073327, at 4300073344 00000100:00000040:3.0:1787697499.941528:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300073327, at 4300073344 00000100:00100000:3.0:1787697500.581474:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697500.581482:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697500.581499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697500.709576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697501.605535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697501.605562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697501.605652:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697501.733586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697502.629474:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697502.629507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697502.629509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697502.757601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697503.141559:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.141570:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697503.141574:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.141584:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5404 00000100:00080000:3.0:1787697503.141593:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697503.141596:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141601:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697503.141606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697503.141608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141612:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697503.141614:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141625:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697503.141645:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697503.141655:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.141673:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.141676:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697503.141678:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.141682:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5404 00000100:00080000:3.0:1787697503.141698:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697503.141702:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141704:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141706:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697503.141709:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697503.141712:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141716:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697503.141718:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697503.141722:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697503.141734:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697503.141735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126221696:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787697503.141739:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697503.141747:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697503.141756:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.141760:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.141765:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.141767:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787697503.141768:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697503.141769:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.141773:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5404 00000100:00080000:3.0:1787697503.141776:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697503.141778:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141780:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141782:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697503.141783:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697503.141786:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141788:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141795:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697503.141798:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697503.141802:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697503.141806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787697503.141809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697503.141814:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697503.141822:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.141825:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.141830:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.141832:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697503.141834:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.141837:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5404 00000100:00080000:3.0:1787697503.141840:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697503.141842:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141844:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141846:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697503.141848:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697503.141851:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697503.141852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697503.141854:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697503.141892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.141903:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697503.141912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697503.141915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697503.141920:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697503.141925:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:2.0:1787697503.141928:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126221696:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697503.141928:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697503.141934:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697503.141935:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126221696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697503.141938:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:2.0:1787697503.141945:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697503.141947:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697503.141959:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697503.141967:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.141968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221824:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697503.141976:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697503.141979:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:2.0:1787697503.141981:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.141981:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697503.141984:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5404 00000100:00080000:3.0:1787697503.141988:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697503.141989:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141990:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.141993:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.141996:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697503.141998:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:2.0:1787697503.142000:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697503.142001:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697503.142003:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697503.142005:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787697503.142007:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697503.142007:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697503.142012:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697503.142016:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:3.0:1787697503.142016:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697503.142022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697503.142023:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697503.142032:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697503.142041:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5414) 00000100:00000040:2.0:1787697503.142046:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697503.142053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697503.142056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697503.142060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697503.142073:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142078:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697503.142087:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142092:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697503.142100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697503.142104:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142110:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697503.142113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697503.142144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697503.142147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697503.142150:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126221952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697503.142170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126221824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697503.142179:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697503.142184:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126221952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697503.142188:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697503.142194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126222080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697503.142210:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142217:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697503.142233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697503.142240:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142250:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697503.142255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697503.142273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697503.142275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697503.142277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126222080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697503.142284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126222208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697503.142296:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142300:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697503.142307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697503.142311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697528 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142335:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697503.142341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697503/real 1787697503] req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697503.142360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697503.142368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697503.142371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697503.142374:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697503.142378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126222208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697503.142382:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126222080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697503.142386:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697503.142390:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126222208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697528 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697503.142394:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697503.653520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697503.653520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697503.653553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697503.781675:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697504.677485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697504.677494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697504.677537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697504.805635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697505.061475:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300078447, at 4300078464 00000100:00000040:2.0:1787697505.061482:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300078447, at 4300078464 00000100:00100000:2.0:1787697505.701524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697505.701528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697505.829569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697506.725532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697506.725532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697506.853590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697507.749521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697507.749521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697507.877560:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697508.261548:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697508.261556:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697508.261559:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697508.261567:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5409 00000100:00080000:2.0:1787697508.261576:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697508.261578:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261582:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261584:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697508.261588:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697508.261596:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261598:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261600:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697508.261602:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261614:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697508.261634:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126222336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697508.261649:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126222336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697508.261668:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697508.261670:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697508.261685:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697508.261688:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5409 00000100:00080000:2.0:1787697508.261691:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697508.261693:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261695:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261696:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697508.261698:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697508.261700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261702:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261703:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697508.261705:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697508.261713:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697508.261721:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697508.261729:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126222464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697508.261739:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126222464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697508.261778:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697508.261783:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697508.261785:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697508.261790:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5409 00000100:00080000:2.0:1787697508.261805:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697508.261809:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261812:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697508.261818:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697508.261831:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261837:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261842:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697508.261844:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697508.261848:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697508.261852:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697508.261860:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126222592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697508.261867:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126222592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697508.261873:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697508.261876:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697508.261877:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697508.261884:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5409 00000100:00080000:2.0:1787697508.261889:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697508.261891:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261893:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261895:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697508.261897:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697508.261900:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261903:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261906:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697508.261908:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697508.261914:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697508.261919:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697508.261928:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126222720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697508.261939:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126222720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697508.261945:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697508.261947:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697508.261949:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697508.261952:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5409 00000100:00080000:2.0:1787697508.261955:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697508.261959:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261964:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261967:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697508.261971:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697508.261975:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697508.261978:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697508.261979:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697508.261981:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697508.261985:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697508.261989:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697508.261998:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126222848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697508.262010:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126222848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697508.262023:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5419) 00000100:00100000:0.0:1787697508.773530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697508.773531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697508.901597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697509.797513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697509.797513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697509.925546:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697510.181495:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300083567, at 4300083584 00000100:00000040:2.0:1787697510.181497:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300083567, at 4300083584 00000100:00100000:0.0:1787697510.821537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697510.821537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697510.949644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697511.845532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697511.845649:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697511.973601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697512.869517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697512.869517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697512.997605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697513.381499:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697513.381508:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697513.381512:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697513.381522:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5414 00000100:00080000:2.0:1787697513.381531:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697513.381534:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381538:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381541:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697513.381546:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697513.381549:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381552:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381555:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697513.381559:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381571:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697513.381596:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126222976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697513.381611:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126222976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697513.381635:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697513.381638:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697513.381654:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697513.381659:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5414 00000100:00080000:2.0:1787697513.381664:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697513.381668:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381671:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381676:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697513.381680:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697513.381689:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381693:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381696:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697513.381702:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697513.381726:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697513.381733:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697513.381746:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126223104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697513.381761:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126223104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697513.381772:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697513.381776:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697513.381778:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697513.381784:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5414 00000100:00080000:2.0:1787697513.381791:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697513.381795:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381799:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381802:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697513.381806:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697513.381818:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381822:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381824:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697513.381828:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697513.381834:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697513.381841:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697513.381857:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126223232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697513.381881:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126223232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697513.381905:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697513.381913:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697513.381917:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697513.381927:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5414 00000100:00080000:2.0:1787697513.381937:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697513.381945:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381950:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381955:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697513.381958:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697513.381961:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.381968:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.381974:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697513.381981:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697513.381988:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697513.381996:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697513.382013:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126223360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697513.382041:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126223360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697513.382056:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697513.382066:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697513.382070:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697513.382076:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5414 00000100:00080000:2.0:1787697513.382084:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697513.382091:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.382099:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.382105:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697513.382111:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697513.382119:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697513.382128:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697513.382133:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697513.382140:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697513.382149:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697513.382156:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697513.382173:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126223488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697513.382191:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126223488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697513.382221:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5424) 00000100:00100000:2.0:1787697513.893525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697513.893573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697514.021637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697514.917505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697514.917505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697515.045612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697515.301527:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300088687, at 4300088704 00000100:00000040:3.0:1787697515.301527:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300088687, at 4300088704 00000100:00100000:2.0:1787697515.941516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697515.941516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697516.069572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697516.965489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697516.965489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697517.093663:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697517.989556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697517.989557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697518.117581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697518.501568:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697518.501581:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697518.501585:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697518.501608:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5419 00000100:00080000:2.0:1787697518.501649:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697518.501653:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501658:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501662:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697518.501668:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697518.501671:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501673:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501676:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697518.501679:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501696:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697518.501725:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126223616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697518.501741:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126223616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697518.501771:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697518.501775:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697518.501795:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697518.501800:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5419 00000100:00080000:2.0:1787697518.501804:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697518.501806:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501809:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501811:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697518.501814:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697518.501817:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501819:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501822:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697518.501824:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697518.501834:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697518.501839:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697518.501849:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126223744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697518.501858:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126223744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697518.501867:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697518.501870:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697518.501871:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697518.501875:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5419 00000100:00080000:2.0:1787697518.501878:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697518.501881:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501883:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501885:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697518.501888:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697518.501897:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501900:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501902:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697518.501905:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697518.501910:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697518.501914:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697518.501924:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126223872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697518.501932:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126223872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697518.501941:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697518.501944:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697518.501945:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697518.501950:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5419 00000100:00080000:2.0:1787697518.501954:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697518.501957:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501962:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697518.501964:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697518.501966:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.501969:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.501971:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697518.501973:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697518.501978:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697518.501984:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697518.501993:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126224000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697518.502007:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126224000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697518.502015:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697518.502018:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697518.502020:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697518.502024:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5419 00000100:00080000:2.0:1787697518.502027:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697518.502029:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.502032:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.502034:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697518.502037:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697518.502039:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697518.502041:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697518.502043:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697518.502046:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697518.502050:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697518.502054:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697518.502063:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126224128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697518.502071:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126224128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697518.502081:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5429) 00000100:00100000:2.0:1787697519.013554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697519.013554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697519.141652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697520.037521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697520.037522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697520.165672:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697520.421543:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300093807, at 4300093824 00000100:00000040:3.0:1787697520.421543:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300093807, at 4300093824 00000100:00100000:3.0:1787697521.061500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697521.061889:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697521.189642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697522.085528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697522.085535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697522.213650:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697523.109546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697523.109549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697523.237646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697523.621568:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697523.621578:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697523.621582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697523.621592:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5424 00000100:00080000:0.0:1787697523.621605:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697523.621607:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621611:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621614:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697523.621626:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697523.621632:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621638:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697523.621641:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621657:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697523.621688:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126224256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697523.621724:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126224256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697523.621751:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697523.621754:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697523.621755:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697523.621759:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5424 00000100:00080000:0.0:1787697523.621763:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697523.621765:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621767:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621769:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697523.621771:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697523.621774:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621775:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621777:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697523.621779:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697523.621788:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697523.621797:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697523.621813:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126224384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697523.621825:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126224384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697523.621835:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697523.621838:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697523.621840:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697523.621850:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5424 00000100:00080000:0.0:1787697523.621856:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697523.621861:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621866:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621870:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697523.621872:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697523.621874:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621875:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621877:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697523.621878:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697523.621883:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697523.621886:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697523.621894:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126224512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697523.621901:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126224512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697523.621906:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697523.621909:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697523.621910:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697523.621913:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5424 00000100:00080000:0.0:1787697523.621915:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697523.621917:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621919:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621920:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697523.621922:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697523.621924:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.621943:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.621946:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697523.621952:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697523.621959:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697523.621968:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697523.621984:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126224640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697523.622000:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126224640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697523.622026:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697523.622032:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697523.622035:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697523.622043:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5424 00000100:00080000:0.0:1787697523.622052:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697523.622060:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.622064:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.622066:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697523.622069:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697523.622074:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697523.622077:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697523.622081:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697523.622087:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697523.622096:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697523.622104:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697523.622117:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126224768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697523.622151:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126224768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697523.622170:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5434) 00000100:00100000:3.0:1787697524.133542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697524.133550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697524.261631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697525.157491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697525.157510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697525.285577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697525.541546:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300098927, at 4300098944 00000100:00000040:2.0:1787697525.541564:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300098927, at 4300098944 00000100:00100000:0.0:1787697526.181529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697526.181531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697526.309605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787697526.693695:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787697526.693742:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787697526.693756:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697526.693796:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126224256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697548 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697526.693813:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697526.693823:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126224384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697548 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697526.693833:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697526.693841:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126224512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697548 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697526.693866:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697526.693875:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126224640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697548 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697526.693884:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697526.693893:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126224768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697548 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697527.205531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697527.205532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697527.333555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697528.229577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697528.229577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697528.229699:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697528.357642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697528.741528:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697528.741538:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697528.741542:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697528.741551:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5429 00000100:00080000:2.0:1787697528.741568:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697528.741571:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741575:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741578:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697528.741583:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697528.741586:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741589:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741592:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697528.741595:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741606:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697528.741629:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126224896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697528.741642:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126224896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697528.741679:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697528.741683:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697528.741685:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697528.741689:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5429 00000100:00080000:2.0:1787697528.741693:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697528.741695:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741698:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741701:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697528.741703:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697528.741707:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741710:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741712:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697528.741715:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697528.741725:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697528.741730:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697528.741748:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126225024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697528.741767:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126225024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697528.741777:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697528.741781:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697528.741783:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697528.741789:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5429 00000100:00080000:2.0:1787697528.741793:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697528.741804:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741810:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697528.741817:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697528.741821:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741824:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741827:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697528.741830:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697528.741836:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697528.741841:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697528.741854:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126225152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697528.741865:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126225152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697528.741880:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697528.741888:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697528.741890:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697528.741894:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5429 00000100:00080000:2.0:1787697528.741898:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697528.741901:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741904:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741906:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697528.741909:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697528.741914:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.741919:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.741925:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697528.741937:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697528.741944:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697528.741958:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697528.741972:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126225280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697528.741995:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126225280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697528.742008:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697528.742013:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697528.742016:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697528.742024:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5429 00000100:00080000:2.0:1787697528.742031:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697528.742036:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.742042:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.742045:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697528.742047:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697528.742050:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697528.742053:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697528.742055:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697528.742061:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697528.742073:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697528.742081:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697528.742095:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126225408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697528.742122:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126225408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697528.742141:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5439) 00000100:00100000:3.0:1787697529.253524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697529.253525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697529.253537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697529.381657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697530.277518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697530.277518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697530.277518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697530.405614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697530.661552:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300104047, at 4300104064 00000100:00000040:3.0:1787697530.661667:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300104047, at 4300104064 00000100:00100000:0.0:1787697531.301514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697531.301514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697531.301514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697531.429594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697532.325493:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697532.325494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697532.325494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697532.453579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697533.349497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697533.349498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697533.349506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697533.477583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697533.861550:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697533.861557:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697533.861561:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697533.861570:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5434 00000100:00080000:2.0:1787697533.861579:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697533.861581:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861585:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861588:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697533.861591:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697533.861593:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861595:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861597:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697533.861599:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861611:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697533.861631:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126225536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697533.861646:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126225536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697533.861662:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697533.861665:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697533.861666:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697533.861669:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5434 00000100:00080000:2.0:1787697533.861671:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697533.861673:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861674:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861676:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697533.861678:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697533.861680:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861683:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861685:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697533.861687:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697533.861696:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697533.861701:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697533.861712:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126225664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697533.861736:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126225664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697533.861747:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697533.861753:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697533.861756:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697533.861760:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5434 00000100:00080000:2.0:1787697533.861763:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697533.861764:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861766:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861768:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697533.861769:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697533.861772:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861774:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861776:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697533.861778:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697533.861782:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697533.861788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697533.861814:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126225792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697533.861827:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126225792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697533.861835:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697533.861841:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697533.861845:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697533.861849:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5434 00000100:00080000:2.0:1787697533.861860:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697533.861867:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861875:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861877:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697533.861880:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697533.861883:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.861887:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.861890:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697533.861893:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697533.861901:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697533.861909:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697533.861923:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126225920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697533.861948:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126225920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697533.861965:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697533.861970:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697533.861973:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697533.861981:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5434 00000100:00080000:2.0:1787697533.861988:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697533.861994:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.862001:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.862006:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697533.862011:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697533.862018:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697533.862024:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697533.862028:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697533.862034:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697533.862043:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697533.862048:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697533.862058:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126226048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697533.862068:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126226048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697533.862083:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5444) 00000100:00100000:0.0:1787697534.373515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697534.373515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697534.373515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697534.501601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697535.397536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697535.397536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697535.525561:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697535.781518:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300109167, at 4300109184 00000100:00000040:2.0:1787697535.781519:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300109167, at 4300109184 00000100:00100000:0.0:1787697536.421500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697536.421500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697536.549608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697537.445513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697537.445513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697537.573583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697538.469481:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697538.469481:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697538.597604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697538.981554:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697538.981562:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697538.981565:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697538.981572:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5439 00000100:00080000:0.0:1787697538.981582:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697538.981585:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981589:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981593:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697538.981597:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697538.981600:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981619:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981622:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697538.981626:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981637:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697538.981661:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126226176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697538.981677:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126226176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697538.981700:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697538.981703:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697538.981705:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697538.981709:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5439 00000100:00080000:0.0:1787697538.981713:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697538.981714:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981716:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981718:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697538.981720:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697538.981722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981726:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697538.981728:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697538.981736:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697538.981740:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697538.981750:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126226304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697538.981778:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126226304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697538.981792:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697538.981795:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697538.981797:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697538.981806:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5439 00000100:00080000:0.0:1787697538.981810:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697538.981813:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981816:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981818:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697538.981821:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697538.981824:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981830:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981837:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697538.981843:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697538.981852:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697538.981860:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697538.981872:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126226432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697538.981884:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126226432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697538.981896:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697538.981902:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697538.981906:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697538.981917:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5439 00000100:00080000:0.0:1787697538.981922:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697538.981943:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981949:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981953:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697538.981957:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697538.981961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.981963:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.981965:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697538.981966:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697538.981971:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697538.981974:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697538.981984:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126226560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697538.982002:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126226560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697538.982026:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697538.982033:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697538.982037:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697538.982046:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5439 00000100:00080000:0.0:1787697538.982053:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697538.982068:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.982077:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.982083:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697538.982086:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697538.982093:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697538.982096:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697538.982098:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697538.982114:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697538.982120:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697538.982125:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697538.982138:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126226688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697538.982169:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126226688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697538.982183:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5450) 00000100:00100000:0.0:1787697539.493497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697539.493497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697539.621543:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697540.517527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697540.517617:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697540.645548:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697540.901512:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300114287, at 4300114304 00000100:00000040:3.0:1787697540.901512:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300114287, at 4300114304 00000100:00100000:0.0:1787697541.541492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697541.541492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697541.669594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697542.565527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697542.565527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697542.565554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697542.693564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697543.589485:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697543.589486:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697543.589499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697543.717631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697544.101692:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.101711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126226816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697544.101740:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.101774:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697544.101816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697544.101837:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.101894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.101926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.101961:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697544.101985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697544.101996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697544.102007:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126226816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697544.102041:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126226816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697544.102058:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697544.102079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126226944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697544.102104:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102111:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697544.102123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697544.102131:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102146:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697544.102152:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697544.102201:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697544.102205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697544.102208:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126226944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697544.102216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697544.102233:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102240:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697544.102252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697544.102258:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102268:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697544.102275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697544.102302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697544.102305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697544.102309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697544.102322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697544.102344:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102351:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697544.102362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697544.102369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102377:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697544.102382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102407:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697544.102414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697544.102417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697544.102442:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697544.102456:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697544.102475:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102482:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697544.102493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697544.102500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697569 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102508:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697544.102513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697544/real 1787697544] req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697544.102533:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697544.102540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697544.102543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697544.102547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697544.102553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697544.102559:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126226944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697544.102566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697544.102572:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126227072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697544.102579:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697544.102584:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126227200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697544.102598:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697544.102604:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126227328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697569 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697544.102610:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697544.613520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697544.613521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697544.613525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697544.741546:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697545.637526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697545.637526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697545.637559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697545.765581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697546.021519:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300119407, at 4300119424 00000100:00000040:3.0:1787697546.021519:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300119407, at 4300119424 00000100:00100000:3.0:1787697546.661529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697546.661542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697546.661614:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697546.789551:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697547.685504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697547.685505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697547.685509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697547.813559:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697548.709513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697548.709514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697548.709514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697548.837595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697549.221680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.221712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126227456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697549.221746:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.221770:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697549.221802:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697549.221819:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.221866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697549/real 1787697549] req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.221895:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.221916:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697549.221931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697549.221947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697549.221956:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.221967:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126227456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697549.221973:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.221979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697549.221989:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.221993:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697549.222001:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697549.222006:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222012:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697549.222015:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697549/real 1787697549] req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222027:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697549.222032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697549.222033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697549.222035:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697549.222043:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126227456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697549.222050:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697549.222063:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126227584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697549.222068:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697549.222076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697549.222114:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222119:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697549.222127:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697549.222131:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222137:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697549.222140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697549/real 1787697549] req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697549.222156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697549.222158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697549.222161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697549.222168:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697549.222183:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222189:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697549.222199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697549.222206:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222214:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697549.222220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697549/real 1787697549] req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697549.222245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697549.222248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697549.222252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697549.222264:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697549.222282:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222290:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697549.222299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697549.222306:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697574 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222313:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697549.222318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697549/real 1787697549] req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697549.222336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697549.222343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697549.222346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697549.222349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697549.222356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126227968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697549.222362:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126227712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697549.222369:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697549.222374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126227840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697549.222381:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697549.222385:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126227968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697574 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697549.222392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697549.733473:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697549.733505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697549.733525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697549.861575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697550.757564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697550.757565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697550.757577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697550.885610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697551.141505:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300124527, at 4300124544 00000100:00000040:3.0:1787697551.141505:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300124527, at 4300124544 00000100:00100000:0.0:1787697551.781507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697551.781507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697551.781519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697551.909584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697552.805519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697552.805519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697552.805523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697552.933574:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697553.829501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697553.829501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697553.829501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697553.957576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697554.341692:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.341711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126228096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697554.341746:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.341768:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697554.341817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697554.341841:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.341888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697554/real 1787697554] req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.341918:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.341941:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697554.341952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697554.341960:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697554.341970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.341990:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126228096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697554.342022:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126228096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697554.342039:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697554.342063:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697554.342126:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342134:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697554.342147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697554.342155:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342165:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697554.342172:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697554/real 1787697554] req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697554.342200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697554.342203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697554.342207:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697554.342220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697554.342241:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342248:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697554.342259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697554.342267:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342281:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697554.342287:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697554/real 1787697554] req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697554.342314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697554.342318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697554.342321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697554.342334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697554.342355:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342362:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697554.342374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697554.342380:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342389:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697554.342395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697554/real 1787697554] req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342413:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697554.342445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697554.342448:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697554.342453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342462:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697554.342467:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126228224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697554.342484:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697554.342489:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126228352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697554.342496:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697554.342501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126228480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697554.342507:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697554.342514:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342523:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697554.342536:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342551:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697554.342564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697554.342570:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697579 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342579:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697554.342584:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697554/real 1787697554] req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342595:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697554.342603:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697554.342611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697554.342614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697554.342617:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697554.342624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126228608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697554.342630:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126228608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697579 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697554.342637:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697554.853512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697554.853512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697554.853512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697554.981569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697555.877508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697555.877508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697555.877508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697556.005582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697556.261539:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300129647, at 4300129664 00000100:00000040:2.0:1787697556.261629:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300129647, at 4300129664 00000100:00100000:2.0:1787697556.901524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697556.901524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697556.901568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697557.029569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697557.925497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697557.925505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697558.053607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697558.949509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697558.949535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697559.077636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697559.461557:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697559.461569:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697559.461575:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697559.461588:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5460 00000100:00080000:2.0:1787697559.461601:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697559.461604:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461609:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461613:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697559.461619:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697559.461623:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461628:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697559.461632:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461651:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697559.461679:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126228736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697559.461696:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126228736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697559.461723:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697559.461728:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697559.461730:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697559.461734:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5460 00000100:00080000:2.0:1787697559.461738:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697559.461741:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461744:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461747:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697559.461750:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697559.461753:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461756:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461759:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697559.461762:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697559.461774:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697559.461781:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697559.461795:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126228864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697559.461807:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126228864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697559.461816:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697559.461820:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697559.461822:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697559.461829:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5460 00000100:00080000:2.0:1787697559.461836:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697559.461843:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461849:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461852:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697559.461855:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697559.461857:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461860:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461863:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697559.461866:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697559.461871:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697559.461876:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697559.461884:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126228992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697559.461902:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126228992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697559.461924:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697559.461928:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697559.461930:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697559.461934:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5460 00000100:00080000:2.0:1787697559.461941:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697559.461944:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461948:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461950:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697559.461953:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697559.461957:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.461963:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.461968:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697559.461974:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697559.461981:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697559.461987:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697559.461997:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126229120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697559.462008:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126229120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697559.462033:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697559.462038:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697559.462040:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697559.462046:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5460 00000100:00080000:2.0:1787697559.462053:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697559.462060:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.462069:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.462073:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697559.462078:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697559.462085:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697559.462093:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697559.462096:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697559.462102:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697559.462109:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697559.462118:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697559.462130:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126229248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697559.462149:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126229248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697559.462167:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5470) 00000100:00100000:2.0:1787697559.973542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697559.973553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697560.101658:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697560.997528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697560.997528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697561.125644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697561.381491:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300134767, at 4300134784 00000100:00000040:3.0:1787697561.381491:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300134767, at 4300134784 00000100:00100000:2.0:1787697562.021535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697562.021638:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697562.149595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697563.045490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697563.045490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697563.173578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697564.069525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697564.069550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697564.197583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697564.581541:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697564.581549:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697564.581553:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697564.581560:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5465 00000100:00080000:3.0:1787697564.581568:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697564.581570:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581573:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581576:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697564.581580:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697564.581582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581584:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581603:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697564.581607:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581617:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697564.581642:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126229376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697564.581656:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126229376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697564.581680:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697564.581684:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697564.581685:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697564.581689:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5465 00000100:00080000:3.0:1787697564.581693:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697564.581696:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581699:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581700:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697564.581703:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697564.581706:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581709:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581711:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697564.581714:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697564.581723:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697564.581729:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697564.581740:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126229504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697564.581762:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126229504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697564.581803:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697564.581809:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697564.581813:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697564.581819:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5465 00000100:00080000:3.0:1787697564.581838:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697564.581841:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581844:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581846:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697564.581849:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697564.581852:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581855:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581858:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697564.581860:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697564.581864:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697564.581868:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697564.581877:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126229632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697564.581885:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126229632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697564.581891:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697564.581895:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697564.581898:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697564.581905:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5465 00000100:00080000:3.0:1787697564.581912:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697564.581919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581921:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581923:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697564.581924:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697564.581927:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.581931:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.581935:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697564.581937:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697564.581941:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697564.581944:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697564.581956:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126229760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697564.581971:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126229760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697564.581983:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697564.581990:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697564.581993:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697564.582000:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5465 00000100:00080000:3.0:1787697564.582004:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697564.582010:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.582015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.582019:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697564.582021:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697564.582027:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697564.582033:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697564.582036:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697564.582045:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697564.582053:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697564.582058:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697564.582070:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126229888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697564.582077:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126229888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697564.582084:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5475) 00000100:00100000:0.0:1787697565.093492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697565.093497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697565.221575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697566.117539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697566.117539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697566.245610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697566.501527:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300139887, at 4300139904 00000100:00000040:3.0:1787697566.501527:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300139887, at 4300139904 00000100:00100000:0.0:1787697567.141521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697567.141647:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697567.269639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697568.165529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697568.165530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697568.293605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697569.189526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697569.189526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697569.317619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697569.701509:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697569.701519:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697569.701522:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697569.701531:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5470 00000100:00080000:3.0:1787697569.701540:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697569.701544:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701547:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701551:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697569.701555:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697569.701558:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701563:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697569.701567:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701592:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697569.701613:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126230016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697569.701628:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126230016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697569.701650:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697569.701654:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697569.701655:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697569.701659:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5470 00000100:00080000:3.0:1787697569.701663:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697569.701665:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701668:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701670:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697569.701673:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697569.701677:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701680:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701683:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697569.701686:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697569.701697:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697569.701703:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697569.701716:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126230144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697569.701726:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126230144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697569.701767:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697569.701777:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697569.701779:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697569.701784:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5470 00000100:00080000:3.0:1787697569.701788:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697569.701791:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701802:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697569.701805:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697569.701809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701816:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701822:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697569.701826:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697569.701831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697569.701836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697569.701846:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126230272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697569.701865:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126230272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697569.701887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697569.701893:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697569.701896:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697569.701904:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5470 00000100:00080000:3.0:1787697569.701912:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697569.701921:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701939:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697569.701943:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697569.701952:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.701959:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.701962:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697569.701965:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697569.701973:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697569.701980:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697569.701995:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126230400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697569.702029:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126230400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697569.702044:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697569.702050:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697569.702054:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697569.702063:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5470 00000100:00080000:3.0:1787697569.702070:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697569.702078:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.702086:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.702090:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697569.702095:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697569.702101:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697569.702110:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697569.702115:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697569.702123:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697569.702131:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697569.702147:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697569.702161:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126230528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697569.702181:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126230528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697569.702192:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5480) 00000100:00100000:3.0:1787697570.213511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697570.213525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697570.341581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697571.237520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697571.237520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697571.365590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697571.621528:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300145007, at 4300145024 00000100:00000040:3.0:1787697571.621528:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300145007, at 4300145024 00000100:00100000:0.0:1787697572.261515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697572.261515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697572.389628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697573.285499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697573.285499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697573.413598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697574.309532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697574.309560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697574.437602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697574.821553:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697574.821563:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697574.821567:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697574.821583:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5475 00000100:00080000:2.0:1787697574.821593:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697574.821597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821602:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821606:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697574.821611:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697574.821614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821616:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821619:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697574.821622:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821634:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697574.821670:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126230656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697574.821684:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126230656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697574.821701:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697574.821703:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697574.821704:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697574.821707:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5475 00000100:00080000:2.0:1787697574.821710:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697574.821712:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821713:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821715:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697574.821717:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697574.821718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821722:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697574.821723:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697574.821731:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697574.821734:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697574.821742:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126230784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697574.821753:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126230784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697574.821777:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697574.821784:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697574.821785:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697574.821788:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5475 00000100:00080000:2.0:1787697574.821790:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697574.821792:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821793:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821795:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697574.821796:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697574.821798:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821800:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821801:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697574.821803:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697574.821806:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697574.821811:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697574.821820:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126230912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697574.821847:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126230912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697574.821858:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697574.821860:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697574.821862:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697574.821864:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5475 00000100:00080000:2.0:1787697574.821867:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697574.821869:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821871:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821872:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697574.821873:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697574.821875:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821876:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821878:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697574.821880:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697574.821882:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697574.821886:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697574.821893:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126231040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697574.821900:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126231040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697574.821905:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697574.821907:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697574.821908:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697574.821910:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5475 00000100:00080000:2.0:1787697574.821912:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697574.821913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821917:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697574.821918:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697574.821922:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697574.821926:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697574.821931:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697574.821935:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697574.821942:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697574.821949:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697574.821957:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126231168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697574.821968:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126231168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697574.821986:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5485) 00000100:00100000:3.0:1787697575.333480:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697575.333503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697575.461605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697576.357529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697576.357529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697576.485544:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697576.741542:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300150127, at 4300150144 00000100:00000040:3.0:1787697576.741542:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300150127, at 4300150144 00000100:00100000:0.0:1787697577.381526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697577.381526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697577.509638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697578.405507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697578.405542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697578.533564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697579.429522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697579.429522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697579.557602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697579.941582:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697579.941593:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697579.941597:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697579.941607:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5480 00000100:00080000:0.0:1787697579.941617:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697579.941619:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941622:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941625:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697579.941629:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697579.941633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941635:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941638:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697579.941641:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941655:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697579.941677:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126231296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697579.941709:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126231296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697579.941726:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697579.941729:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697579.941730:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697579.941733:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5480 00000100:00080000:0.0:1787697579.941736:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697579.941738:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941740:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941741:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697579.941744:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697579.941746:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941748:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941749:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697579.941751:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697579.941759:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697579.941762:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697579.941772:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126231424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697579.941798:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126231424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697579.941806:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697579.941812:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697579.941814:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697579.941822:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5480 00000100:00080000:0.0:1787697579.941825:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697579.941827:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941829:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941830:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697579.941832:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697579.941834:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941837:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941840:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697579.941845:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697579.941853:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697579.941860:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697579.941874:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126231552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697579.941884:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126231552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697579.941893:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697579.941899:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697579.941901:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697579.941906:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5480 00000100:00080000:0.0:1787697579.941912:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697579.941915:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941919:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941921:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697579.941924:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697579.941930:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.941941:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.941944:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697579.941947:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697579.941952:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697579.941957:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697579.941967:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126231680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697579.941993:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126231680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697579.941999:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697579.942003:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697579.942004:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697579.942010:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5480 00000100:00080000:0.0:1787697579.942014:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697579.942019:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.942021:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.942024:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697579.942026:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697579.942031:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697579.942038:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697579.942044:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697579.942048:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697579.942057:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697579.942066:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697579.942078:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126231808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697579.942114:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126231808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697579.942125:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5491) 00000100:00100000:2.0:1787697580.453530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697580.453530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697580.581533:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697581.477540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697581.477541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697581.605590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697581.861535:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300155247, at 4300155264 00000100:00000040:2.0:1787697581.861537:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300155247, at 4300155264 00000100:00100000:2.0:1787697582.501524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697582.501533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697582.629663:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697583.525574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697583.525574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697583.653609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697584.549573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697584.549574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697584.677686:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697585.061563:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697585.061578:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697585.061584:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697585.061597:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5486 00000100:00080000:0.0:1787697585.061610:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697585.061614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061622:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697585.061628:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697585.061631:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061633:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061636:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697585.061639:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061657:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697585.061689:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126231936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697585.061706:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126231936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.061733:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697585.061736:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697585.061737:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697585.061740:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5486 00000100:00080000:0.0:1787697585.061762:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697585.061766:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061770:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061772:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697585.061776:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697585.061779:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061782:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061785:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697585.061792:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697585.061806:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697585.061813:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697585.061830:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126232064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697585.061841:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126232064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.061854:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697585.061863:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697585.061867:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697585.061875:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5486 00000100:00080000:0.0:1787697585.061880:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697585.061882:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061894:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061896:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697585.061899:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697585.061902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061905:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061913:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697585.061920:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697585.061927:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697585.061931:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697585.061941:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126232192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697585.061949:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126232192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.061957:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697585.061964:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697585.061968:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697585.061972:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5486 00000100:00080000:0.0:1787697585.061977:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697585.061980:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061983:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.061985:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697585.061988:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697585.061991:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.061995:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.062000:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697585.062006:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697585.062011:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697585.062021:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697585.062035:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126232320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697585.062096:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126232320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697585.062111:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697585.062120:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697585.062125:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697585.062134:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5486 00000100:00080000:0.0:1787697585.062140:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697585.062149:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.062153:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.062155:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697585.062160:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697585.062164:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697585.062183:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697585.062189:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697585.062196:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697585.062205:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697585.062211:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697585.062222:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126232448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697585.062234:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126232448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697585.062243:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5496) 00000100:00100000:3.0:1787697585.573562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697585.573704:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697585.701610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697586.597539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697586.597540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697586.725657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697586.981573:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300160367, at 4300160384 00000100:00000040:3.0:1787697586.981573:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300160367, at 4300160384 00000100:00100000:0.0:1787697587.621554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697587.621554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697587.749650:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697588.645544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697588.645545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697588.773615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697589.669544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697589.669544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697589.797603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697590.181586:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697590.181596:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697590.181600:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697590.181609:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5491 00000100:00080000:3.0:1787697590.181619:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697590.181622:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.181626:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.181629:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697590.181634:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697590.181638:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.181640:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.181643:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697590.181646:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697590.181660:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697590.181693:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126232576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697590.181708:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126232576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697590.181737:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697590.181741:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697590.181743:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697590.181747:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5491 00000100:00080000:3.0:1787697590.181751:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697590.181753:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.181756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.181772:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697590.181775:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697590.181779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.181783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.181785:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697590.181788:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697590.181798:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697590.181811:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697590.181831:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126232704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697590.181853:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126232704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697590.181894:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697590.181899:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697590.181901:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697590.181908:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5491 00000100:00080000:3.0:1787697590.181920:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697590.181925:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.181928:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.181930:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697590.181933:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697590.181936:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.181938:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.181940:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697590.181943:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697590.181949:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697590.181963:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697590.181984:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126232832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697590.181997:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126232832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697590.182005:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697590.182009:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697590.182018:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697590.182023:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5491 00000100:00080000:3.0:1787697590.182028:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697590.182032:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.182034:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.182037:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697590.182039:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697590.182042:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.182045:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.182047:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697590.182049:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697590.182054:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697590.182059:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697590.182069:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126232960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697590.182081:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126232960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697590.182089:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697590.182097:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697590.182102:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697590.182114:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5491 00000100:00080000:3.0:1787697590.182121:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697590.182132:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.182145:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.182149:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697590.182152:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697590.182155:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697590.182158:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697590.182164:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697590.182174:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697590.182187:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697590.182197:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697590.182215:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126233088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697590.182477:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126233088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697590.182494:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5501) 00000100:00100000:3.0:1787697590.693528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697590.693559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697590.821581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697591.717520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697591.717534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697591.845601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697592.101505:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300165487, at 4300165504 00000100:00000040:2.0:1787697592.101507:0:88278:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300165487, at 4300165504 00000100:00100000:0.0:1787697592.741514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697592.741514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697592.869649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787697593.253675:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787697593.253720:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697593.253737:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697593.253778:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126232576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697615 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697593.253794:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697593.253802:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126232704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697615 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697593.253811:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697593.253818:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126232832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697615 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697593.253827:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697593.253835:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126232960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697615 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697593.253844:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697593.253852:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126233088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697615 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697593.765524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697593.765524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697593.896521:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697594.789494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697594.789494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697594.917567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697595.301637:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697595.301648:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697595.301665:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697595.301675:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5496 00000100:00080000:3.0:1787697595.301687:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697595.301692:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.301696:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.301699:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697595.301703:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697595.301707:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.301709:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.301712:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697595.301715:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697595.301726:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697595.301759:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126233216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697595.301779:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126233216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697595.301800:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697595.301803:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697595.301805:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697595.301809:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5496 00000100:00080000:3.0:1787697595.301812:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697595.301815:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.301817:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.301819:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697595.301821:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697595.301824:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.301826:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.301834:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697595.301837:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697595.301846:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697595.301852:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697595.301877:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126233344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697595.301910:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126233344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697595.301929:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697595.301933:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697595.301935:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697595.301940:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5496 00000100:00080000:3.0:1787697595.301944:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697595.301957:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.301969:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.301974:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697595.301979:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697595.301982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.301984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.301986:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697595.301988:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697595.301993:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697595.301998:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697595.302011:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126233472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697595.302074:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126233472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697595.302083:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697595.302089:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697595.302093:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697595.302103:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5496 00000100:00080000:3.0:1787697595.302108:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697595.302111:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.302117:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.302120:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697595.302125:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697595.302132:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.302137:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.302141:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697595.302148:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697595.302158:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697595.302164:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697595.302180:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126233600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697595.302208:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126233600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697595.302219:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697595.302230:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697595.302232:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697595.302242:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5496 00000100:00080000:3.0:1787697595.302247:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697595.302253:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.302256:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.302258:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697595.302261:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697595.302265:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697595.302270:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697595.302275:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697595.304448:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697595.304459:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697595.304469:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697595.304487:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126233728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697595.304499:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126233728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697595.304526:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5506) 00000100:00100000:3.0:1787697595.813546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697595.813554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697595.941643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697596.837486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697596.837498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787697596.965549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697597.221503:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300170607, at 4300170624 00000100:00000040:3.0:1787697597.221521:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300170607, at 4300170624 00000100:00100000:0.0:1787697597.861524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697597.861524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697597.989532:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697598.885468:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697598.885495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787697599.013611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697599.909569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697599.910531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:2.0:1787697600.421561:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697600.421572:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697600.421576:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697600.421587:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5501 00000100:00080000:2.0:1787697600.421599:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697600.421602:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421605:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421609:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697600.421613:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697600.421616:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421617:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421620:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697600.421622:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421640:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697600.421676:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126233856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697600.421694:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126233856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697600.421714:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697600.421718:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697600.421719:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697600.421722:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5501 00000100:00080000:2.0:1787697600.421725:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697600.421727:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421729:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421730:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697600.421732:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697600.421734:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421736:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421737:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697600.421739:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697600.421749:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697600.421753:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697600.421760:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126233984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697600.421767:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126233984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697600.421777:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697600.421780:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697600.421782:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697600.421787:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5501 00000100:00080000:2.0:1787697600.421790:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697600.421801:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421811:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421821:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697600.421825:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697600.421832:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421840:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421842:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697600.421845:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697600.421852:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697600.421860:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697600.421877:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126234112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697600.421887:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126234112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697600.421901:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697600.421909:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697600.421912:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697600.421918:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5501 00000100:00080000:2.0:1787697600.421923:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697600.421928:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421936:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421945:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697600.421954:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697600.421959:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.421962:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.421963:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697600.421966:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697600.421972:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697600.421977:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697600.421989:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126234240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697600.422001:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126234240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697600.422082:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697600.422085:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697600.422087:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697600.422092:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5501 00000100:00080000:2.0:1787697600.422098:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697600.422109:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.422123:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.422127:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697600.422132:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697600.422138:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697600.422147:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697600.422156:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697600.422178:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697600.422190:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697600.422198:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697600.422223:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126234368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697600.422234:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126234368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697600.422256:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5511) 00000100:00100000:2.0:1787697600.933562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697600.933711:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697601.957494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697601.957496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697602.341474:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300175727, at 4300175744 00000100:00000040:2.0:1787697602.341491:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300175727, at 4300175744 00000100:00100000:0.0:1787697602.981523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697602.981523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697604.005533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697604.005711:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697605.029525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697605.029525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697605.541558:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697605.541579:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697605.541584:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697605.541598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5506 00000100:00080000:2.0:1787697605.541611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697605.541615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.541620:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.541624:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697605.541630:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697605.541634:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.541637:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.541640:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697605.541644:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697605.541663:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697605.541696:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126234496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697605.541716:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126234496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697605.541733:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697605.541737:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697605.541739:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697605.541743:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5506 00000100:00080000:2.0:1787697605.541747:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697605.541750:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.541753:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.541756:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697605.541759:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697605.541762:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.541765:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.541768:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697605.541770:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697605.541791:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697605.541797:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697605.541810:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126234624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697605.541820:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126234624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697605.541838:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697605.541849:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697605.541854:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697605.541868:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5506 00000100:00080000:2.0:1787697605.541878:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697605.541887:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.541902:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.541907:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697605.541910:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697605.541913:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.541916:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.541918:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697605.541922:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697605.541928:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697605.541934:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697605.541952:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126234752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697605.541970:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126234752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697605.541990:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697605.541994:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697605.541997:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697605.542003:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5506 00000100:00080000:2.0:1787697605.542014:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697605.542018:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.542029:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.542034:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697605.542036:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697605.542041:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.542046:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.542049:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697605.542053:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697605.542059:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697605.542069:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697605.542089:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126234880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697605.542100:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126234880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697605.542110:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697605.542137:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697605.542142:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697605.542149:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5506 00000100:00080000:2.0:1787697605.542157:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697605.542166:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.542178:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.542185:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697605.542189:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697605.542203:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697605.542221:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697605.542230:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697605.542239:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697605.542251:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697605.542263:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697605.542281:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126235008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697605.542302:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126235008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697605.542322:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5516) 00000100:00100000:0.0:1787697606.053596:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697606.053596:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697607.077539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697607.077656:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697607.461507:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300180847, at 4300180864 00000100:00000040:3.0:1787697607.461507:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300180847, at 4300180864 00000100:00100000:0.0:1787697608.101539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697608.101548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697609.125547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697609.125547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697610.149551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697610.149885:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697610.661538:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697610.661548:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697610.661552:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697610.661561:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5511 00000100:00080000:3.0:1787697610.661571:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697610.661574:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661578:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661581:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697610.661585:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697610.661588:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661591:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661593:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697610.661596:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661611:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697610.661635:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126235136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697610.661651:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126235136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697610.661677:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697610.661680:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697610.661682:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697610.661686:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5511 00000100:00080000:3.0:1787697610.661690:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697610.661693:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661696:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661711:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697610.661714:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697610.661718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661721:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661724:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697610.661727:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697610.661737:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697610.661744:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697610.661760:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126235264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697610.661777:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126235264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697610.661786:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697610.661792:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697610.661794:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697610.661803:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5511 00000100:00080000:3.0:1787697610.661810:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697610.661816:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661820:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661822:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697610.661825:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697610.661828:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661831:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661833:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697610.661836:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697610.661841:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697610.661852:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697610.661864:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126235392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697610.661887:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126235392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697610.661895:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697610.661899:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697610.661901:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697610.661907:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5511 00000100:00080000:3.0:1787697610.661914:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697610.661919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661924:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697610.661928:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697610.661935:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.661942:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.661948:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697610.661954:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697610.661964:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697610.661973:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697610.661989:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126235520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697610.662003:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126235520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697610.662024:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697610.662046:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697610.662049:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697610.662055:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5511 00000100:00080000:3.0:1787697610.662062:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697610.662070:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.662074:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.662077:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697610.662080:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697610.662083:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697610.662090:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697610.662094:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697610.662101:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697610.662109:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697610.662117:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697610.662133:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126235648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697610.662167:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126235648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697610.662185:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5521) 00000100:00100000:0.0:1787697611.173494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697611.173494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697612.197510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697612.197510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697612.581552:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300185967, at 4300185984 00000100:00000040:2.0:1787697612.581558:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300185967, at 4300185984 00000100:00100000:3.0:1787697613.221538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697613.221540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697614.245488:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697614.245536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697615.269520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697615.269534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697615.781558:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697615.781568:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697615.781572:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697615.781582:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5516 00000100:00080000:0.0:1787697615.781594:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697615.781605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.781609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.781612:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697615.781617:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697615.781620:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.781622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.781625:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697615.781628:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697615.781642:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697615.781665:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126235776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697615.781742:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126235776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697615.781765:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697615.781769:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697615.781771:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697615.781775:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5516 00000100:00080000:0.0:1787697615.781779:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697615.781782:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.781785:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.781787:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697615.781789:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697615.781792:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.781795:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.781798:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697615.781801:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697615.781810:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697615.781814:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697615.781828:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126235904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697615.781855:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126235904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697615.781868:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697615.781872:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697615.781874:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697615.781886:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5516 00000100:00080000:0.0:1787697615.781892:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697615.781899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.781907:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.781912:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697615.781915:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697615.781918:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.781921:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.781924:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697615.781926:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697615.781932:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697615.781962:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697615.781975:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126236032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697615.781992:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126236032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697615.782003:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697615.782006:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697615.782007:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697615.782011:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5516 00000100:00080000:0.0:1787697615.782016:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697615.782019:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.782024:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.782027:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697615.782030:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697615.782036:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.782046:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.782050:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697615.782056:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697615.782067:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697615.782074:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697615.782091:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126236160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697615.782109:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126236160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697615.782127:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697615.782135:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697615.782138:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697615.782144:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5516 00000100:00080000:0.0:1787697615.782149:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697615.782151:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.782155:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.782159:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697615.782163:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697615.782170:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697615.782177:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697615.782180:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697615.782183:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697615.782188:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697615.782196:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697615.782210:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126236288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697615.782234:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126236288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697615.782251:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5526) 00000100:00100000:2.0:1787697616.293532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697616.293532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697617.317512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697617.317512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697617.701533:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300191087, at 4300191104 00000100:00000040:3.0:1787697617.701533:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300191087, at 4300191104 00000100:00100000:2.0:1787697618.341520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697618.341520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697619.365497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697619.365497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697620.389581:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697620.389583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697620.901543:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697620.901553:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697620.901557:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697620.901566:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5521 00000100:00080000:3.0:1787697620.901576:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697620.901580:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901584:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901587:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697620.901592:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697620.901595:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901598:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901601:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697620.901604:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901616:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697620.901638:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126236416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697620.901654:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126236416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697620.901679:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697620.901683:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697620.901684:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697620.901688:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5521 00000100:00080000:3.0:1787697620.901692:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697620.901707:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901709:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901712:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697620.901715:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697620.901717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901721:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697620.901723:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697620.901730:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697620.901733:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697620.901743:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126236544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697620.901770:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126236544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697620.901780:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697620.901786:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697620.901787:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697620.901790:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5521 00000100:00080000:3.0:1787697620.901793:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697620.901794:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901796:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901797:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697620.901799:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697620.901801:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901802:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901804:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697620.901813:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697620.901819:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697620.901825:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697620.901835:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126236672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697620.901842:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126236672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697620.901877:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697620.901882:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697620.901885:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697620.901892:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5521 00000100:00080000:3.0:1787697620.901896:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697620.901900:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901902:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901904:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697620.901907:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697620.901909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901915:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697620.901917:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697620.901921:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697620.901924:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697620.901931:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126236800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697620.901937:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126236800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697620.901948:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697620.901950:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697620.901952:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697620.901955:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5521 00000100:00080000:3.0:1787697620.901958:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697620.901962:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901967:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901971:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697620.901974:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697620.901979:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697620.901982:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697620.901983:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697620.901985:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697620.901989:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697620.901992:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697620.901998:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126236928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697620.902009:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126236928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697620.902040:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5532) 00000100:00100000:2.0:1787697621.413504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697621.413544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697622.437506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697622.437506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:2.0:1787697622.821516:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300196207, at 4300196224 00000100:00000040:3.0:1787697622.821516:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300196207, at 4300196224 00000100:00100000:0.0:1787697623.461506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697623.461517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697624.485513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697624.485513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697625.509535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697625.509535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:3.0:1787697626.021544:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.021571:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697626.021575:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.021586:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5527 00000100:00080000:3.0:1787697626.021596:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697626.021600:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021604:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021608:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697626.021613:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697626.021616:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021619:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021621:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697626.021624:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021635:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697626.021658:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126237056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697626.021673:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126237056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697626.021695:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.021699:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697626.021700:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.021705:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5527 00000100:00080000:3.0:1787697626.021708:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697626.021711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021717:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697626.021720:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697626.021729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021733:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021735:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697626.021740:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697626.021749:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697626.021757:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697626.021797:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126237184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697626.021812:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126237184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697626.021822:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.021827:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697626.021830:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.021838:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5527 00000100:00080000:3.0:1787697626.021846:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697626.021851:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021854:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021857:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697626.021859:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697626.021862:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021865:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021868:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697626.021871:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697626.021876:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697626.021882:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697626.021894:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126237312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697626.021925:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126237312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697626.021934:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.021937:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697626.021939:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.021945:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5527 00000100:00080000:3.0:1787697626.021952:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697626.021958:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021963:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021965:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697626.021968:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697626.021973:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.021980:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.021984:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697626.021990:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697626.021998:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697626.022006:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697626.022020:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126237440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697626.022049:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126237440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697626.022062:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697626.022069:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697626.022073:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697626.022085:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5527 00000100:00080000:3.0:1787697626.022090:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697626.022094:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.022099:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.022104:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697626.022108:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697626.022115:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697626.022121:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697626.022127:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697626.022131:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697626.022137:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697626.022143:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697626.022158:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126237568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697626.022172:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126237568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697626.022189:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5537) 00000100:00100000:3.0:1787697626.533518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697626.533555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697627.557518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697627.557519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697627.941537:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300201327, at 4300201344 00000100:00000040:3.0:1787697627.941537:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300201327, at 4300201344 00000100:00100000:0.0:1787697628.581526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697628.581542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697629.605519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697629.605519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697630.629530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697630.629540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:2.0:1787697631.141567:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697631.141577:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697631.141581:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697631.141591:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5532 00000100:00080000:2.0:1787697631.141604:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697631.141607:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141613:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697631.141618:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697631.141620:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141624:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697631.141627:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141640:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697631.141666:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126237696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697631.141692:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126237696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697631.141710:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697631.141713:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697631.141714:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697631.141718:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5532 00000100:00080000:2.0:1787697631.141720:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697631.141722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141725:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697631.141727:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697631.141729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141731:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141733:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697631.141735:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697631.141745:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697631.141749:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697631.141760:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126237824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697631.141783:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126237824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697631.141791:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697631.141796:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697631.141798:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697631.141802:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5532 00000100:00080000:2.0:1787697631.141804:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697631.141806:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141807:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141808:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697631.141810:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697631.141811:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141812:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141814:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697631.141816:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697631.141819:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697631.141822:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697631.141832:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126237952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697631.141858:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126237952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697631.141872:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697631.141874:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697631.141875:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697631.141877:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5532 00000100:00080000:2.0:1787697631.141879:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697631.141881:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141883:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141884:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697631.141885:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697631.141887:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141888:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141889:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697631.141891:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697631.141894:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697631.141898:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697631.141902:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126238080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697631.141907:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126238080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697631.141918:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697631.141921:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697631.141922:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697631.141925:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5532 00000100:00080000:2.0:1787697631.141928:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697631.141933:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141938:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141941:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697631.141944:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697631.141947:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697631.141950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697631.141951:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697631.141953:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697631.141956:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697631.141959:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697631.141964:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126238208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697631.141969:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126238208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697631.141978:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5542) 00000100:00100000:0.0:1787697631.653537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697631.653542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697632.677525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697632.677525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697633.061562:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300206447, at 4300206464 00000100:00000040:3.0:1787697633.061562:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300206447, at 4300206464 00000100:00100000:2.0:1787697633.701564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697633.701681:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697634.725512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697634.725512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697635.749503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697635.749581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:2.0:1787697636.261532:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697636.261540:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697636.261543:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697636.261551:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5537 00000100:00080000:2.0:1787697636.261561:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697636.261563:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261566:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261568:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697636.261572:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697636.261574:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261576:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261578:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697636.261580:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261600:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697636.261620:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126238336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697636.261630:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126238336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697636.261645:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697636.261647:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697636.261649:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697636.261651:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5537 00000100:00080000:2.0:1787697636.261654:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697636.261655:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261657:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261658:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697636.261661:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697636.261662:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261664:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261665:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697636.261667:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697636.261674:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697636.261678:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697636.261684:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126238464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697636.261691:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126238464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697636.261700:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697636.261707:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697636.261709:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697636.261716:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5537 00000100:00080000:2.0:1787697636.261721:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697636.261725:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261731:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261737:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697636.261740:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697636.261742:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261745:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261747:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697636.261750:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697636.261755:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697636.261762:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697636.261779:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126238592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697636.261803:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126238592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697636.261813:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697636.261820:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697636.261822:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697636.261827:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5537 00000100:00080000:2.0:1787697636.261832:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697636.261839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261847:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261852:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697636.261855:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697636.261859:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261862:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261864:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697636.261867:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697636.261874:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697636.261879:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697636.261891:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126238720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697636.261914:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126238720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697636.261921:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697636.261930:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697636.261933:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697636.261939:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5537 00000100:00080000:2.0:1787697636.261945:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697636.261950:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261955:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261959:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697636.261963:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697636.261968:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697636.261974:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697636.261977:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697636.261982:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697636.261987:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697636.261993:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697636.262003:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126238848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697636.262023:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126238848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697636.262030:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5547) 00000100:00100000:3.0:1787697636.773549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697636.773555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697637.797556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697637.797556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:2.0:1787697638.181526:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300211567, at 4300211584 00000100:00000040:3.0:1787697638.181526:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300211567, at 4300211584 00000100:00100000:2.0:1787697638.821509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697638.821511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697639.845533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697639.845533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697640.869559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697640.869564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:0.0:1787697641.381580:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697641.381592:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697641.381596:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697641.381607:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5542 00000100:00080000:0.0:1787697641.381623:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697641.381626:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381630:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381633:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697641.381638:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697641.381640:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381642:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381666:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697641.381669:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381680:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697641.381706:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126238976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697641.381719:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126238976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697641.381745:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697641.381750:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697641.381752:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697641.381756:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5542 00000100:00080000:0.0:1787697641.381761:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697641.381764:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381767:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381770:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697641.381777:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697641.381780:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381782:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381783:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697641.381785:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697641.381795:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697641.381799:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697641.381812:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126239104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697641.381828:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126239104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697641.381844:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697641.381847:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697641.381848:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697641.381859:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5542 00000100:00080000:0.0:1787697641.381867:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697641.381871:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381873:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381874:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697641.381877:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697641.381879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381881:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381882:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697641.381884:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697641.381887:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697641.381891:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697641.381898:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126239232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697641.381906:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126239232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697641.381915:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697641.381918:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697641.381919:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697641.381922:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5542 00000100:00080000:0.0:1787697641.381925:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697641.381935:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381942:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697641.381944:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697641.381947:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.381948:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.381949:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697641.381951:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697641.381954:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697641.381957:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697641.381965:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126239360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697641.381976:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126239360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697641.381986:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697641.381988:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697641.381989:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697641.381993:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5542 00000100:00080000:0.0:1787697641.381996:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697641.382000:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.382004:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.382007:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697641.382010:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697641.382014:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697641.382019:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697641.382020:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697641.382028:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697641.382034:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697641.382039:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697641.382047:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126239488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697641.382059:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126239488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697641.382067:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5552) 00000100:00100000:3.0:1787697641.893554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697641.893560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697642.917687:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697642.917688:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:2.0:1787697643.301551:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300216687, at 4300216704 00000100:00000040:3.0:1787697643.301551:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300216687, at 4300216704 00000100:00100000:2.0:1787697643.941567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697643.941696:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697644.965557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697644.965557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697645.989568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697645.989622:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697646.501551:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697646.501578:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697646.501585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697646.501601:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5547 00000100:00080000:2.0:1787697646.501614:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697646.501618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501627:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697646.501633:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697646.501636:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501638:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501642:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697646.501644:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501662:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697646.501692:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126239616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697646.501708:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126239616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697646.501730:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697646.501734:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697646.501736:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697646.501741:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5547 00000100:00080000:2.0:1787697646.501744:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697646.501747:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501750:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501752:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697646.501755:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697646.501758:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501761:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501763:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697646.501766:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697646.501776:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697646.501781:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697646.501790:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126239744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697646.501799:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126239744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697646.501806:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697646.501809:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697646.501811:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697646.501814:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5547 00000100:00080000:2.0:1787697646.501817:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697646.501820:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501823:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501826:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697646.501828:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697646.501830:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501833:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501835:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697646.501837:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697646.501840:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697646.501844:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697646.501850:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126239872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697646.501863:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126239872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697646.501870:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697646.501873:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697646.501875:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697646.501878:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5547 00000100:00080000:2.0:1787697646.501881:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697646.501883:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501885:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501887:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697646.501889:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697646.501892:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501894:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501896:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697646.501899:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697646.501904:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697646.501907:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697646.501915:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126240000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697646.501924:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126240000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697646.501931:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697646.501935:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697646.501937:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697646.501941:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5547 00000100:00080000:2.0:1787697646.501945:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697646.501947:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501952:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697646.501954:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697646.501957:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697646.501960:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697646.501962:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697646.501964:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697646.501968:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697646.501971:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697646.501980:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126240128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697646.501990:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126240128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697646.502000:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5557) 00000100:00100000:2.0:1787697647.013522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697647.013522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697648.037537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697648.037538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697648.421509:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300221807, at 4300221824 00000100:00000040:3.0:1787697648.421510:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300221807, at 4300221824 00000100:00100000:0.0:1787697649.061639:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697649.061655:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697650.085553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697650.085577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697651.109532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697651.109532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697651.621793:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.621819:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126240256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697651.621856:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.621876:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697651.621922:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697651.621944:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.621989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697651/real 1787697651] req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622034:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697651.622045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697651.622053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697651.622061:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126240256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697651.622091:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126240256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697651.622110:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697651.622130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697651.622168:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622188:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697651.622204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697651.622214:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622229:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697651.622237:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697651/real 1787697651] req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697651.622266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697651.622269:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697651.622273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697651.622287:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697651.622306:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622314:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697651.622354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697651.622360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622370:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697651.622376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697651/real 1787697651] req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697651.622406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697651.622409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697651.622412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697651.622444:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622453:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697651.622466:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622474:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697651.622483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697651.622490:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622506:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697651.622511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697651/real 1787697651] req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697651.622540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697651.622543:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697651.622547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697651.622561:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126240384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697651.622568:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697651.622574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126240512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697651.622580:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697651.622586:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126240640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697651.622594:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697651.622601:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697651.622621:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622629:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697651.622639:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697651.622645:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697676 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622654:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697651.622660:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697651/real 1787697651] req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697651.622681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697651.622689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697651.622691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697651.622695:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697651.622703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126240768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697651.622709:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126240768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697676 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697651.622716:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697652.133539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697652.133539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697653.157479:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697653.157485:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:2.0:1787697653.541529:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300226927, at 4300226944 00000100:00000040:3.0:1787697653.541529:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300226927, at 4300226944 00000100:00100000:2.0:1787697654.181534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697654.181534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697655.205574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697655.205574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697656.229573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697656.229575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00080000:3.0:1787697656.741579:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697656.741592:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697656.741597:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697656.741610:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5557 00000100:00080000:3.0:1787697656.741624:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697656.741627:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.741630:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741633:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697656.741638:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697656.741640:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.741642:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741645:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697656.741675:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741691:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697656.741723:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697656.741740:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697656.741760:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697656.741768:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697656.741770:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697656.741775:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5557 00000100:00080000:3.0:1787697656.741779:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697656.741782:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.741785:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741787:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697656.741790:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697656.741793:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.741797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741799:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697656.741802:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697656.741814:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697656.741820:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697656.741822:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697656.741832:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697656.741846:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697656.741852:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126240896:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697656.741874:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697656.741877:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697656.741879:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697656.741883:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5557 00000100:00080000:3.0:1787697656.741887:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697656.741893:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697656.741893:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.741899:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741903:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697656.741907:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697656.741913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697656.741917:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697656.741921:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741925:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697656.741927:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697656.741933:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697656.741938:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697656.741949:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697656.741959:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697656.741968:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697656.741971:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697656.741973:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697656.741977:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5557 00000100:00080000:3.0:1787697656.741981:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697656.741990:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.741993:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.741995:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697656.741997:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697656.742000:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.742010:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.742013:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697656.742017:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697656.742022:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697656.742029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697656.742029:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697656.742045:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241024:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787697656.742045:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697656.742062:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697656.742071:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697656.742074:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697656.742083:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697656.742086:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697656.742092:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697656.742095:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5557 00000100:00080000:3.0:1787697656.742101:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697656.742104:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697656.742108:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.742110:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787697656.742112:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:3.0:1787697656.742113:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697656.742120:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697656.742125:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697656.742126:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697656.742130:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787697656.742140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241152:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697656.742143:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697656.742151:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697656.742157:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697656.742158:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697656.742169:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697656.742172:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697656.742185:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787697656.742186:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697656.742201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697656.742202:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5567) 00000100:00100000:2.0:1787697656.742214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697656.742227:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697656.742249:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697656.742262:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:2.0:1787697656.742271:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697656.742281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697656.742292:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697656.742300:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697656.742311:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697657.253569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697657.253569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697657.253595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697658.277534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697658.277535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697658.277555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00000040:3.0:1787697658.661509:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300232047, at 4300232064 00000100:00000040:2.0:1787697658.661510:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300232047, at 4300232064 00000100:00100000:0.0:1787697659.301561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697659.301561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697659.301568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000400:00000100:0.0:1787697659.813678:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787697659.813724:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697659.813741:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697659.813801:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697659.813822:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697659.813833:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697659.813845:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697659.813853:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697659.813863:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697659.813871:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697659.813896:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697659.813906:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697681 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697656/real 1787697659] req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697659.814041:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697659.814057:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697659.814063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697659.814067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241408:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787697659.814082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697656/real 1787697659] req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697659.814100:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697659.814107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697659.814111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697659.814115:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241280:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787697659.814127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697656/real 1787697659] req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814137:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697659.814151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697659.814157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697659.814161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697659.814165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241024:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787697659.814194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697656/real 1787697659] req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697659.814213:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697659.814221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697659.814224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697659.814228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126241152:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787697659.814240:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697656/real 1787697659] req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814257:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697659.814265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697659.814273:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697659.814276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697659.814279:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697659.814286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126240896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697659.814292:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126241408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697659.814306:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697659.814316:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126241280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697659.814323:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697659.814328:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126241024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697659.814335:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697659.814339:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126241152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697659.814346:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697659.814351:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126240896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697681 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697659.814358:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697660.325497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697660.325497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697660.325497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697661.349540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697661.349541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697661.349543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697661.861523:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697661.861535:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697661.861540:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697661.861553:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5562 00000100:00080000:2.0:1787697661.861565:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697661.861569:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861573:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861577:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697661.861583:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697661.861586:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861589:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861592:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697661.861595:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861608:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697661.861638:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126241536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697661.861654:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126241536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697661.861683:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697661.861687:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697661.861689:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697661.861694:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5562 00000100:00080000:2.0:1787697661.861699:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697661.861702:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861705:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861707:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697661.861710:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697661.861714:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861717:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861719:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697661.861722:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697661.861734:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697661.861742:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697661.861755:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126241664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697661.861771:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126241664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697661.861781:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697661.861784:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697661.861786:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697661.861790:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5562 00000100:00080000:2.0:1787697661.861793:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697661.861796:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861799:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861803:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697661.861806:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697661.861813:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861819:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861824:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697661.861830:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697661.861835:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697661.861840:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697661.861848:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126241792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697661.861863:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126241792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697661.861872:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697661.861875:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697661.861879:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697661.861887:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5562 00000100:00080000:2.0:1787697661.861893:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697661.861897:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861900:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861903:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697661.861907:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697661.861913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.861919:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.861924:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697661.861928:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697661.861933:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697661.861940:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697661.861966:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126241920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697661.861989:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126241920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697661.862009:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697661.862016:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697661.862019:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697661.862025:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5562 00000100:00080000:2.0:1787697661.862030:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697661.862037:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.862041:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.862043:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697661.862047:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697661.862052:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697661.862059:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697661.862063:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697661.862069:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697661.862076:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697661.862084:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697661.862096:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126242048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697661.862124:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126242048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697661.862137:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5572) 00000100:00100000:0.0:1787697662.373525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697662.373540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697662.373614:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697663.397541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697663.397541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697663.397543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:3.0:1787697663.781572:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300237167, at 4300237184 00000100:00000040:2.0:1787697663.781774:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300237167, at 4300237184 00000100:00100000:0.0:1787697664.421530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697664.421530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697665.445474:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697665.445485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697666.469551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697666.469551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00080000:2.0:1787697666.981514:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697666.981530:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697666.981534:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697666.981544:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5567 00000100:00080000:2.0:1787697666.981554:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697666.981558:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981562:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981566:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697666.981571:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697666.981574:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981577:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981579:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697666.981582:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981595:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697666.981618:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126242176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697666.981632:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126242176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697666.981653:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697666.981657:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697666.981659:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697666.981663:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5567 00000100:00080000:2.0:1787697666.981666:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697666.981669:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981672:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981674:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697666.981678:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697666.981681:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981684:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981694:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697666.981702:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697666.981714:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697666.981721:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697666.981733:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126242304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697666.981760:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126242304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697666.981772:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697666.981775:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697666.981777:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697666.981782:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5567 00000100:00080000:2.0:1787697666.981786:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697666.981793:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981800:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697666.981803:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697666.981806:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981812:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981816:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697666.981821:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697666.981828:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697666.981834:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697666.981845:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126242432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697666.981859:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126242432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697666.981872:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697666.981879:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697666.981882:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697666.981889:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5567 00000100:00080000:2.0:1787697666.981896:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697666.981902:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981915:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697666.981919:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697666.981926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.981933:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.981939:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697666.981944:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697666.981951:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697666.981957:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697666.981970:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126242560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697666.981985:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126242560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697666.982001:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697666.982017:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697666.982022:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697666.982027:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5567 00000100:00080000:2.0:1787697666.982033:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697666.982041:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.982047:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.982051:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697666.982068:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697666.982077:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697666.982083:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697666.982088:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697666.982095:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697666.982104:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697666.982111:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697666.982127:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126242688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697666.982151:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126242688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697666.982169:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5578) 00000100:00100000:0.0:1787697667.493527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697667.493527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697668.517520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697668.517520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697668.901519:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300242287, at 4300242304 00000100:00000040:3.0:1787697668.901519:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300242287, at 4300242304 00000100:00100000:0.0:1787697669.541544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697669.541544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697670.565504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697670.565504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697671.589511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697671.589525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00000040:2.0:1787697672.101682:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.101697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126242816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697672.101722:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.101732:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697672.101763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697672.101775:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.101800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.101829:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.101841:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697672.101852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697672.101859:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697672.101866:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.101881:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126242816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697672.101891:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126242816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697672.101899:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697672.101913:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.101932:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126242944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697672.101953:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.101977:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697672.101989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697672.101998:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102009:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697672.102016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697672.102021:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697672.102027:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697672.102035:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697672.102040:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697672.102041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697672.102050:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697672.102054:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697672.102062:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697672.102069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000040:1.0:1787697672.102073:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5583) 00000100:00080000:2.0:1787697672.102076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697672.102082:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126242944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697672.102099:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697672.102117:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102124:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697672.102135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697672.102141:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102153:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697672.102158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697672.102183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697672.102185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697672.102189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697672.102199:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126242944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697672.102206:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697672.102211:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126243072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697672.102218:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697672.102225:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697672.102251:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102257:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697672.102265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697672.102270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102275:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697672.102278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697672.102294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697672.102296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697672.102298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697672.102305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697672.102316:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102323:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697672.102328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697672.102331:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697697 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102337:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697672.102339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697672/real 1787697672] req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697672.102350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697672.102354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697672.102355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697672.102357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697672.102361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697672.102364:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126243200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697672.102369:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697672.102372:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126243328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697697 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697672.102375:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697672.613480:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697672.613483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697672.613487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697672.741672:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697673.637526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697673.637548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697673.637548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697673.765571:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697674.021535:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300247407, at 4300247424 00000100:00000040:2.0:1787697674.021536:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300247407, at 4300247424 00000100:00100000:3.0:1787697674.661507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697674.661512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697674.661537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697674.789616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697675.685552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697675.685558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697675.685588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697675.813550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697676.709537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697676.709546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697676.709547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697676.837569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697677.221542:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697677.221552:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697677.221557:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697677.221567:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5578 00000100:00080000:0.0:1787697677.221577:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697677.221581:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.221585:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.221589:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697677.221593:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697677.221597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.221600:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.221603:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697677.221611:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697677.221624:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697677.221662:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697677.221678:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697677.221701:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697677.221705:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697677.221706:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697677.221710:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5578 00000100:00080000:0.0:1787697677.221715:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697677.221718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.221722:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.221725:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697677.221728:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697677.221733:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697677.221734:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697677.221738:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.221743:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697677.221749:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697677.221755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126243456:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787697677.221759:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697677.221766:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697677.221779:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697677.221780:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697677.221810:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.221810:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697677.221821:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697677.221825:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697677.221832:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697677.221837:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5578 00000100:00080000:0.0:1787697677.221842:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787697677.221846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697677.221849:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.221855:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.221857:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697677.221860:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697677.221864:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697677.221867:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.221874:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.221891:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697677.221894:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697677.221900:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697677.221905:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697677.221911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697677/real 1787697677] req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697677.221917:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697677.221935:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.221943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697677.221948:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697677.221955:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:2.0:1787697677.221955:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697677.221959:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697677.221967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697677.221968:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5578 00000100:00080000:2.0:1787697677.221973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787697677.221975:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697677.221977:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697677.221982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.221989:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697677.221991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126243456:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697677.221994:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697677.221998:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697677.221999:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126243456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697677.222006:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697677.222011:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.222015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.222017:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697677.222020:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697677.222027:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697677.222029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697677.222034:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697677.222039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243584:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697677.222049:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697677.222053:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222073:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697677.222076:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697677.222086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697677.222095:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697677.222097:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697677.222101:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697677.222105:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697677.222112:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:0.0:1787697677.222113:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5578 00000100:00080000:0.0:1787697677.222120:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697677.222122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697677/real 1787697677] req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697677.222127:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.222133:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.222138:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697677.222142:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:2.0:1787697677.222147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697677.222148:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697677.222155:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697677.222159:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697677.222165:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697677.222169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697677.222172:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697677.222177:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787697677.222180:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787697677.222181:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697677.222187:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697677.222195:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697677.222199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697677.222206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697677.222228:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222235:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697677.222245:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5588) 00000100:00000040:2.0:1787697677.222245:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697677.222260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697677.222270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222279:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697677.222285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697677/real 1787697677] req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697677.222307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697677.222309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697677.222311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697677.222319:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126243584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697677.222324:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697677.222328:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126243712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697677.222332:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697677.222337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697677.222352:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222357:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697677.222364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697677.222368:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222373:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697677.222376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697677/real 1787697677] req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222389:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697677.222396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697677.222400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697677.222403:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697677.222416:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697677.222465:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222469:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697677.222477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697677.222481:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697702 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222486:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697677.222489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697677/real 1787697677] req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697677.222501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697677.222505:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697677.222507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697677.222509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697677.222514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126243968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697677.222517:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126243840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697677.222521:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697677.222524:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126243968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697702 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697677.222528:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697677.733500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697677.733500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697677.733503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697677.861580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697678.757522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697678.757522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697678.757551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697678.885652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697679.141528:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300252527, at 4300252544 00000100:00000040:3.0:1787697679.141549:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300252527, at 4300252544 00000100:00100000:2.0:1787697679.781504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697679.781519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697679.781540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697679.909589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697680.805539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697680.805539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697680.805547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697680.933622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697681.829479:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697681.829508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697681.829509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697681.957575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697682.341551:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697682.341561:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697682.341565:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697682.341577:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5583 00000100:00080000:1.0:1787697682.341588:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697682.341591:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341595:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341597:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697682.341602:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697682.341605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341607:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341609:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697682.341612:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341630:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697682.341656:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697682.341668:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697682.341685:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697682.341688:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697682.341689:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697682.341692:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5583 00000100:00080000:1.0:1787697682.341694:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697682.341696:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341697:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341698:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697682.341700:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697682.341702:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341704:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341706:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697682.341708:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697682.341716:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697682.341721:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697682.341727:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697682.341735:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697682.341745:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697682.341747:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697682.341749:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697682.341754:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697682.341768:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5583 00000100:00080000:1.0:1787697682.341776:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697682.341779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341781:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697682.341797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126244096:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787697682.341798:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697682.341803:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697682.341810:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341813:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341815:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697682.341819:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697682.341827:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697682.341834:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697682.341834:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.341846:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787697682.341855:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697682.341867:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697682.341879:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787697682.341879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787697682.341887:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697682.341891:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697682.341901:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5583 00000100:00000040:2.0:1787697682.341901:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697682.341917:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697682.341924:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341931:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697682.341934:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787697682.341939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697682/real 1787697682] req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697682.341942:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.341958:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.341967:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697682.341978:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697682.341981:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697682.341992:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697682.341995:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697682.342006:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787697682.342008:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697682.342017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697682.342021:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697682.342028:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697682.342033:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126244096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697682.342051:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126244096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697682.342064:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697682.342066:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787697682.342080:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697682.342085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697682.342090:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697682.342096:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697682.342103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244224:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787697682.342110:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5583 00000100:00080000:1.0:1787697682.342119:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697682.342123:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697682.342130:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697682.342144:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697682.342144:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787697682.342153:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697682.342156:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:2.0:1787697682.342162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697682.342167:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697682.342172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697682.342178:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697682.342186:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787697682.342187:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:1.0:1787697682.342201:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697682.342201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697682/real 1787697682] req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697682.342217:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697682.342221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697682.342227:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697682.342231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697682.342238:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697682.342242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00100000:1.0:1787697682.342248:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697682.342249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697682.342313:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697682.342334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697682.342355:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5593) 00000100:00000040:2.0:1787697682.342365:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342371:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697682.342378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697682.342385:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342392:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697682.342396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697682/real 1787697682] req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697682.342417:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697682.342447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697682.342452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697682.342466:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697682.342487:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342492:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697682.342500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697682.342504:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342510:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697682.342514:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697682/real 1787697682] req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697682.342534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697682.342535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697682.342537:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697682.342544:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126244224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697682.342549:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697682.342553:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126244352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697682.342557:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697682.342560:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126244480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697682.342564:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697682.342569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342574:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697682.342581:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342586:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697682.342591:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697682.342594:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697707 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342599:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697682.342602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697682/real 1787697682] req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697682.342613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697682.342617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697682.342619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697682.342621:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697682.342624:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697682.342627:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126244608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697707 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697682.342631:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697682.853494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697682.853498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697682.853518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697682.981619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697683.877534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697683.877540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697683.877600:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697684.005613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697684.261571:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300257647, at 4300257664 00000100:00000040:3.0:1787697684.261574:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300257647, at 4300257664 00000100:00100000:3.0:1787697684.901511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697684.901513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697684.901525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697685.029534:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697685.925508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697685.925521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697685.925548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697686.053584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697686.949513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697686.949513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697686.949513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697687.077587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697687.461504:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697687.461513:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697687.461517:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697687.461527:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5588 00000100:00080000:1.0:1787697687.461537:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697687.461541:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461545:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461548:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697687.461553:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697687.461556:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461559:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461561:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697687.461565:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461577:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697687.461600:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697687.461614:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697687.461637:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697687.461641:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697687.461643:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697687.461647:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5588 00000100:00080000:1.0:1787697687.461650:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697687.461653:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461656:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461658:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697687.461661:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697687.461665:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461668:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461670:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697687.461673:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697687.461681:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697687.461684:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697687.461687:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697687.461712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126244736:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787697687.461726:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697687.461737:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.461743:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697687.461752:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697687.461756:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697687.461758:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697687.461762:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5588 00000100:00080000:1.0:1787697687.461768:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697687.461769:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461775:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461781:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697687.461784:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697687.461787:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461789:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461791:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697687.461794:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697687.461799:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697687.461804:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787697687.461809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787697687.461816:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697687.461829:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697687.461830:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697687.461847:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697687.461854:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697687.461857:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697687.461862:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5588 00000100:00080000:1.0:1787697687.461866:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697687.461868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461872:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461874:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697687.461877:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787697687.461882:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697687/real 1787697687] req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697687.461883:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461889:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.461894:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697687.461898:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697687.461904:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697687.461904:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697687.461913:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697687.461915:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697687.461922:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787697687.461925:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697687.461927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697687.461934:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697687.461948:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.461948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126244736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697687.461959:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126244736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697687.461967:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697687.461975:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697687.461979:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:2.0:1787697687.461984:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787697687.461985:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5588 00000100:00080000:1.0:1787697687.461990:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697687.461994:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697687.461997:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.462000:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697687.462002:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697687.462006:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697687.462008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697687.462012:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697687.462019:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787697687.462023:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244864:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787697687.462027:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697687.462033:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697687.462038:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697687.462039:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462048:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787697687.462052:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697687.462061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697687.462066:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462069:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462094:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787697687.462095:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5598) 00000100:00100000:2.0:1787697687.462099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697687/real 1787697687] req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462114:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697687.462118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697687.462120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697687.462122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697687.462131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462136:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697687.462142:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462147:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697687.462152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697687.462161:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462167:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697687.462170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697687/real 1787697687] req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697687.462186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697687.462187:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697687.462189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126244992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697687.462196:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697687.462209:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462213:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697687.462218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697687.462222:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462227:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697687.462229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697687/real 1787697687] req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697687.462251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697687.462253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697687.462255:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697687.462261:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126244864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697687.462266:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697687.462270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126244992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697687.462274:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697687.462277:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126245120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697687.462281:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697687.462285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462289:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697687.462295:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462299:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697687.462305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697687.462309:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697712 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462314:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697687.462317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697687/real 1787697687] req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697687.462328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697687.462332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697687.462333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697687.462336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697687.462340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697687.462343:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126245248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697712 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697687.462347:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697687.973526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697687.973526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697687.973527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697688.101590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697688.997487:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697688.997487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697688.997490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697689.125550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697689.381487:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300262767, at 4300262784 00000100:00000040:3.0:1787697689.381487:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300262767, at 4300262784 00000100:00100000:0.0:1787697690.021512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697690.021513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697690.021513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697690.149584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697691.045516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697691.045516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697691.045516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697691.173549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697692.069514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697692.069541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697692.069541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697692.197654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697692.581558:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697692.581565:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697692.581569:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697692.581576:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5593 00000100:00080000:2.0:1787697692.581584:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697692.581592:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581595:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581597:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697692.581601:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697692.581603:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581604:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581606:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697692.581608:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581631:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697692.581651:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697692.581661:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697692.581688:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697692.581691:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697692.581693:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697692.581697:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5593 00000100:00080000:2.0:1787697692.581701:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697692.581704:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581712:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697692.581715:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697692.581719:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581722:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581724:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697692.581727:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697692.581736:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697692.581742:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697692.581743:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697692.581762:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697692.581763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126245376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697692.581782:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697692.581792:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697692.581794:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697692.581795:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697692.581798:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5593 00000100:00080000:2.0:1787697692.581802:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697692.581805:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697692.581809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581820:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581830:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697692.581838:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697692.581845:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581847:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581851:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581853:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697692.581854:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697692.581859:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697692.581863:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697692.581871:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697692.581878:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697692.581884:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697692.581886:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787697692.581892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697692.581892:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697692.581901:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5593 00000100:00080000:2.0:1787697692.581904:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697692.581907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581909:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581910:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697692.581912:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697692.581914:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581916:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581918:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787697692.581924:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697692.581924:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697692.581932:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697692.581940:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697692.581948:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697692.581961:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697692.581966:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697692.581968:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697692.581969:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697692.581971:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5593 00000100:00080000:2.0:1787697692.581973:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697692.581974:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581978:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581979:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697692.581981:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697692.581984:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697692.581988:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697692.581991:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787697692.581999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697692/real 1787697692] req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697692.582000:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697692.582010:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697692.582018:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697692.582026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697692.582031:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697692.582056:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697692.582062:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697692.582086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787697692.582092:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5603) 00000100:00080000:1.0:1787697692.582098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697692.582107:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126245376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697692.582127:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126245376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697692.582140:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697692.582160:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697692.582187:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582195:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697692.582219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697692.582228:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582240:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697692.582246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697692/real 1787697692] req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697692.582285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697692.582289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697692.582293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697692.582307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697692.582329:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582337:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697692.582356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697692.582364:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582379:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697692.582386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697692/real 1787697692] req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697692.582414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697692.582440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697692.582445:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697692.582460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582469:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697692.582485:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582492:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697692.582504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697692.582511:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582521:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697692.582527:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697692/real 1787697692] req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582548:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697692.582556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697692.582560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697692.582565:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582572:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697692.582579:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697692.582599:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582606:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697692.582656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697692.582664:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697717 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582673:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697692.582679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697692/real 1787697692] req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582698:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697692.582706:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697692.582713:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697692.582717:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697692.582721:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697692.582729:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126245888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697692.582734:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126245504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697692.582743:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697692.582747:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126245632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697692.582752:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697692.582755:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126245760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697692.582759:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697692.582762:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126245888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697717 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697692.582766:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697693.093540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697693.093540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697693.093601:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697693.221556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697694.117533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697694.117533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697694.117533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697694.245633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697694.501539:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300267887, at 4300267904 00000100:00000040:3.0:1787697694.501539:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300267887, at 4300267904 00000100:00100000:0.0:1787697695.141523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697695.141523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697695.141529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697695.269548:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697696.165559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697696.165559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697696.165572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697696.293603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697697.189528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697697.189541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697697.189543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697697.317603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697697.701585:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697697.701599:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697697.701604:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697697.701618:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5598 00000100:00080000:1.0:1787697697.701631:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697697.701635:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701646:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701650:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697697.701656:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697697.701658:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701660:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701663:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697697.701665:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701680:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697697.701705:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697697.701718:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697697.701739:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697697.701743:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697697.701745:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697697.701748:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5598 00000100:00080000:1.0:1787697697.701751:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697697.701754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701758:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697697.701760:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697697.701762:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701763:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701765:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697697.701767:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697697.701773:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697697.701777:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697697.701784:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697697.701791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126246016:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787697697.701793:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697697.701801:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697697.701807:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697697.701809:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697697.701810:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697697.701813:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5598 00000100:00080000:1.0:1787697697.701815:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697697.701817:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701819:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701821:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697697.701823:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697697.701827:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697697.701827:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697697.701833:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701837:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697697.701842:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697697.701846:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787697697.701847:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697697.701852:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697697.701859:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697697.701866:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697697.701871:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697697.701873:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697697.701874:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697697.701877:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5598 00000100:00080000:1.0:1787697697.701879:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697697.701881:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701883:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697697.701885:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:2.0:1787697697.701888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701891:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701894:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701896:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697697.701898:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697697.701903:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697697.701907:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697697.701910:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697697.701924:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697697.701936:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.701945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697697/real 1787697697] req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697697.701946:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697697.701955:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697697.701959:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697697.701965:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5598 00000100:00100000:2.0:1787697697.701967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697697.701970:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697697.701978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.701986:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697697.701989:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697697.701991:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697697.701994:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697697.701999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697697.702003:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697697.702008:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787697697.702013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787697697.702014:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697697.702019:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787697697.702023:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:1.0:1787697697.702030:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697697.702033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697697.702042:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126246016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697697.702061:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5608) 00000100:00000040:2.0:1787697697.702067:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126246016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697697.702077:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697697.702092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697697.702109:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702114:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697697.702123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697697.702127:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702134:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697697.702138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697697/real 1787697697] req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697697.702156:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697697.702157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697697.702161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697697.702168:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697697.702179:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702185:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697697.702195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697697.702200:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702209:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697697.702214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697697/real 1787697697] req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697697.702239:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697697.702241:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697697.702244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697697.702256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697697.702270:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702275:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697697.702282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697697.702286:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702292:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697697.702295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697697/real 1787697697] req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697697.702312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697697.702314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697697.702316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697697.702324:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697697.702335:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702339:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697697.702344:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697697.702347:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697722 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702352:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697697.702355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697697/real 1787697697] req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697697.702367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697697.702371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697697.702372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697697.702374:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697697.702378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697697.702381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126246144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697697.702389:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697697.702393:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126246272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697697.702398:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697697.702401:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126246400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697697.702406:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697697.702408:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126246528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697722 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697697.702412:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697698.213557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697698.213564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697698.213564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697698.341576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697699.237511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697699.237535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697699.237536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697699.365585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697699.621477:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300273007, at 4300273024 00000100:00000040:2.0:1787697699.621540:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300273007, at 4300273024 00000100:00100000:1.0:1787697700.261557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697700.261559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697700.261591:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697700.389609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697701.285495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697701.285495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697701.285496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697701.413591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697702.309559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697702.309559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697702.309561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697702.437598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697702.821520:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697702.821527:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697702.821531:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697702.821543:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5603 00000100:00080000:0.0:1787697702.821552:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697702.821570:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821573:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821576:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697702.821579:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697702.821582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821583:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821586:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697702.821588:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821596:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697702.821619:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697702.821634:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697702.821658:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697702.821662:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697702.821664:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697702.821668:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5603 00000100:00080000:0.0:1787697702.821672:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697702.821674:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821678:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821680:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697702.821684:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697702.821687:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821689:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821690:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697702.821696:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697702.821698:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697702.821704:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697702.821713:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697702.821721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126246656:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787697702.821723:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697702.821737:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697702.821743:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697702.821746:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697702.821747:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697702.821749:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5603 00000100:00080000:0.0:1787697702.821752:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697702.821754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821757:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697702.821759:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697702.821763:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697702.821771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697702.821776:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821779:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821782:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697702.821784:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697702.821787:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697702.821791:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697702.821797:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697702.821803:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697702.821811:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787697702.821816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697702.821822:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697702.821825:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697702.821830:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5603 00000100:00080000:0.0:1787697702.821836:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697702.821839:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697702.821842:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821854:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697702.821859:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697702.821865:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821867:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821869:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697702.821871:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697702.821875:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697702.821878:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697702.821892:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697702.821892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697702/real 1787697702] req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697702.821906:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697702.821918:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697702.821920:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697702.821921:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697702.821925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697702.821927:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5603 00000100:00080000:0.0:1787697702.821934:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697702.821939:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697702.821940:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697702.821948:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697702.821951:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697702.821955:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787697702.821959:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787697702.821961:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697702.821965:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:2.0:1787697702.821970:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126246656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697702.821975:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697702.821979:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126246656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697702.821983:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697702.821987:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697702.821992:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697702.821994:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:0.0:1787697702.821997:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697702.822003:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697702.822009:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697702.822012:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697702.822031:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697702.822042:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5613) 00000100:00000040:2.0:1787697702.822054:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822063:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697702.822074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697702.822079:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822089:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697702.822093:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697702/real 1787697702] req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822105:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697702.822110:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697702.822111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697702.822113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697702.822121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697702.822133:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822137:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697702.822142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697702.822146:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822151:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697702.822154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697702/real 1787697702] req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697702.822170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697702.822171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697702.822173:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126246912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697702.822180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697702.822190:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822194:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697702.822199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697702.822202:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822207:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697702.822210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697702/real 1787697702] req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697702.822228:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697702.822230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697702.822232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697702.822239:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126246784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697702.822243:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697702.822247:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126246912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697702.822251:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697702.822254:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126247040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697702.822258:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697702.822262:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697702.822273:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822278:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697702.822283:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697702.822286:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697727 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822291:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697702.822294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697702/real 1787697702] req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697702.822305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697702.822309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697702.822310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697702.822312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697702.822316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697702.822319:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126247168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697727 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697702.822323:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697703.333521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697703.333521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697703.333524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697703.461578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697704.357507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697704.357507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697704.357507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697704.485579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697704.741494:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300278127, at 4300278144 00000100:00000040:3.0:1787697704.741494:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300278127, at 4300278144 00000100:00100000:3.0:1787697705.381489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697705.381517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697705.381521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697705.509620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697706.405547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697706.405547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697706.405553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697706.533600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697707.429478:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697707.429487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697707.429544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697707.557606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697707.941527:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697707.941535:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697707.941539:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697707.941546:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5608 00000100:00080000:2.0:1787697707.941555:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697707.941557:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941560:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941562:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697707.941566:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697707.941568:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941569:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941572:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697707.941574:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941584:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697707.941604:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697707.941614:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697707.941638:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697707.941641:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697707.941643:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697707.941646:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5608 00000100:00080000:2.0:1787697707.941648:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697707.941650:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941652:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941653:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697707.941655:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697707.941657:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941660:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941663:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697707.941666:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697707.941676:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697707.941679:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697707.941682:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697707.941696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126247296:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787697707.941699:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697707.941708:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697707.941714:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697707.941716:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697707.941718:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697707.941721:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5608 00000100:00000040:1.0:1787697707.941723:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697707.941725:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697707.941732:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697707.941738:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941738:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941743:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697707.941745:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697707.941747:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941749:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941750:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697707.941753:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697707.941756:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697707.941761:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697707.941771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787697707.941773:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697707.941786:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697707.941788:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697707.941806:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697707.941809:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697707.941810:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697707.941813:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5608 00000100:00080000:2.0:1787697707.941817:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697707.941819:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941822:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941823:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697707.941825:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697707.941830:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697707.941831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697707/real 1787697707] req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697707.941832:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941836:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697707.941840:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697707.941847:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697707.941853:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697707.941864:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697707.941870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697707.941875:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697707.941893:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697707.941896:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697707.941897:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697707.941900:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697707.941907:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5608 00000100:00080000:2.0:1787697707.941911:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697707.941912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787697707.941914:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697707.941920:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697707.941920:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941925:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787697707.941929:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697707.941929:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697707.941935:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941941:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697707.941945:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787697707.941949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126247296:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787697707.941950:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697707.941957:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697707.941959:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126247296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697707.941961:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697707.941971:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787697707.941972:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697707.941981:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.941989:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697707.941990:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5619) 00000100:00100000:1.0:1787697707.941998:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697707.942010:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942015:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697707.942024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697707.942028:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942035:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697707.942039:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697707/real 1787697707] req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697707.942056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697707.942058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697707.942060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697707.942067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697707.942078:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942082:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697707.942088:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697707.942091:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942096:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697707.942099:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697707/real 1787697707] req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942114:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697707.942118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697707.942119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697707.942121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697707.942128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697707.942140:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942144:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697707.942149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697707.942152:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942157:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697707.942160:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697707/real 1787697707] req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942170:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942175:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697707.942179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697707.942180:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697707.942182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697707.942188:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126247424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697707.942193:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697707.942196:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126247552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697707.942200:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697707.942203:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126247680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697707.942207:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697707.942211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697707.942222:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942227:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697707.942232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697707.942235:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697732 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942240:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697707.942243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697707/real 1787697707] req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942249:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697707.942254:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697707.942257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697707.942259:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697707.942261:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697707.942264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126247808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697707.942267:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126247808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697732 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697707.942271:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697708.453571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697708.453571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697708.453572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697708.581633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697709.477506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697709.477506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697709.477506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697709.605604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697709.861512:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300283247, at 4300283264 00000100:00000040:2.0:1787697709.861514:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300283247, at 4300283264 00000100:00100000:0.0:1787697710.501510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697710.501510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697710.501510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697710.629584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697711.525485:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697711.525487:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697711.525496:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697711.653580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697712.549513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697712.549513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697712.549598:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697712.677638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697713.061542:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697713.061552:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697713.061555:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697713.061566:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5614 00000100:00080000:2.0:1787697713.061576:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697713.061580:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061584:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061587:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697713.061592:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697713.061595:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061598:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061601:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697713.061604:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061617:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697713.061643:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697713.061657:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697713.061680:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697713.061684:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697713.061686:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697713.061690:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5614 00000100:00080000:2.0:1787697713.061693:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697713.061696:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061699:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061701:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697713.061704:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697713.061706:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061709:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061711:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697713.061713:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697713.061721:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697713.061725:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697713.061734:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697713.061745:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697713.061754:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697713.061757:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697713.061760:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697713.061764:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5614 00000100:00080000:2.0:1787697713.061768:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697713.061770:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061774:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061776:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697713.061779:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697713.061782:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061787:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061795:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787697713.061796:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697713.061802:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697713.061813:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697713.061822:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697713.061826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126247936:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787697713.061841:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697713.061850:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697713.061858:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697713.061860:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697713.061862:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697713.061867:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5614 00000100:00000040:1.0:1787697713.061868:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697713.061872:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697713.061886:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697713.061896:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061897:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061907:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697713.061910:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697713.061913:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061915:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061916:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697713.061919:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697713.061925:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697713.061928:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697713.061935:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697713.061941:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697713.061947:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697713.061953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697713.061960:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697713.061963:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697713.061966:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5614 00000100:00080000:2.0:1787697713.061971:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697713.061975:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.061978:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.061980:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697713.061982:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697713.061987:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697713.061990:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697713.062004:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697713.062014:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697713.062024:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697713.062029:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697713.062033:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697713.062044:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697713.062054:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697713.062090:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5624) 00000100:00100000:1.0:1787697713.062102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697713.062119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697713.062122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697713.062128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126247936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697713.062141:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697713.062159:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062164:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697713.062172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697713.062176:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062184:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697713.062188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697713.062206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697713.062208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697713.062211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697713.062219:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126247936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697713.062227:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697713.062236:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126248064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697713.062241:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697713.062254:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697713.062311:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062319:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697713.062330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697713.062338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062348:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697713.062355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062378:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697713.062387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697713.062389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697713.062392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697713.062402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697713.062459:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062464:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697713.062474:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697713.062479:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062486:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697713.062490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697713.062509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697713.062511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697713.062513:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697713.062521:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697713.062536:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062541:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697713.062546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697713.062550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697738 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062556:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697713.062559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697713/real 1787697713] req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697713.062572:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697713.062576:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697713.062578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697713.062580:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697713.062586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697713.062590:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126248192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697713.062595:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697713.062600:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126248320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697713.062605:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697713.062608:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126248448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697738 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697713.062613:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697713.573497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697713.573497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697713.573501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697713.701624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697714.597515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697714.597515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697714.597554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697714.725569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697714.981512:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300288367, at 4300288384 00000100:00000040:3.0:1787697714.981512:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300288367, at 4300288384 00000100:00100000:2.0:1787697715.621532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697715.621532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697715.621536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697715.749565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697716.645523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697716.645523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697716.645523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697716.773544:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697717.669523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697717.669523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697717.669524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697717.797587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697718.181558:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697718.181570:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697718.181574:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697718.181585:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5619 00000100:00080000:0.0:1787697718.181603:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697718.181606:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.181610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181613:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697718.181635:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697718.181640:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.181651:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181655:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697718.181658:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181671:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697718.181693:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697718.181704:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697718.181721:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697718.181723:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697718.181725:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697718.181729:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5619 00000100:00080000:0.0:1787697718.181734:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697718.181737:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.181740:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181742:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697718.181746:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697718.181749:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.181752:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181755:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697718.181758:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697718.181768:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697718.181774:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697718.181781:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697718.181791:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697718.181803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126248576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697718.181819:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697718.181831:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697718.181835:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697718.181838:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697718.181841:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697718.181845:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5619 00000100:00080000:0.0:1787697718.181853:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697718.181859:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697718.181862:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697718.181867:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181871:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697718.181873:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697718.181877:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.181880:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181883:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697718.181886:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697718.181892:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697718.181899:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697718.181908:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697718.181912:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697718.181925:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.181931:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697718.181939:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697718.181947:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697718.181955:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697718.181962:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5619 00000100:00080000:0.0:1787697718.181967:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697718.181976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697718/real 1787697718] req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697718.181984:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.181992:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.181996:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697718.181999:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697718.182003:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697718.182008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697718.182010:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.182016:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697718.182022:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697718.182031:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697718.182037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697718.182039:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697718.182053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697718.182057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787697718.182063:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697718.182066:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697718.182097:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126248576:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697718.182114:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697718.182119:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697718.182123:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697718.182123:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126248576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697718.182138:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5619 00000020:00000040:1.0:1787697718.182139:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787697718.182145:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697718.182149:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.182153:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.182156:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697718.182159:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697718.182162:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697718.182189:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697718.182194:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697718.182197:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697718.182200:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697718.182205:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697718.182213:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697718.182219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248704:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697718.182227:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697718.182241:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697718.182243:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182268:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697718.182272:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5629) 00000800:00000100:1.0:1787697718.182286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697718.182298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182310:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697718.182316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697718/real 1787697718] req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182347:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697718.182355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697718.182359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697718.182364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697718.182379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697718.182403:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182411:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697718.182451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697718.182460:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182470:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697718.182477:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697718/real 1787697718] req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697718.182517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697718.182521:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697718.182526:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182535:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697718.182541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697718.182566:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182575:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697718.182588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697718.182595:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182606:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697718.182612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697718/real 1787697718] req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182654:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697718.182663:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697718.182666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697718.182671:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182680:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126248960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697718.182686:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126248704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697718.182696:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697718.182702:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126248832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697718.182711:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697718.182716:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126248960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697718.182724:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697718.182732:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182742:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697718.182757:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182765:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697718.182778:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697718.182787:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697743 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182797:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697718.182804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697718/real 1787697718] req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182817:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697718.182827:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697718.182836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697718.182840:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697718.182844:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697718.182853:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697718.182859:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126249088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697743 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697718.182868:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697718.693489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697718.693494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697718.693519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697718.821585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697719.717536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697719.717540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697719.717540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697719.845600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697720.101509:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300293487, at 4300293504 00000100:00000040:3.0:1787697720.101510:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300293487, at 4300293504 00000100:00100000:1.0:1787697720.741522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697720.741555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697720.741580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697720.869672:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697721.765546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697721.765560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697721.765726:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697721.893606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697722.789530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697722.789530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697722.789546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697722.917571:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697723.301549:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697723.301562:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697723.301566:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697723.301577:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5624 00000100:00080000:0.0:1787697723.301589:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697723.301591:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301594:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301597:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697723.301602:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697723.301604:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301606:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301608:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697723.301611:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301643:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697723.301669:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697723.301682:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697723.301702:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697723.301704:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697723.301705:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697723.301708:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5624 00000100:00080000:0.0:1787697723.301711:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697723.301713:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301714:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301716:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697723.301718:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697723.301720:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301721:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301722:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697723.301724:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697723.301731:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697723.301735:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697723.301740:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697723.301747:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697723.301760:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697723.301764:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697723.301767:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697723.301770:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697723.301776:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5624 00000100:00080000:0.0:1787697723.301781:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697723.301784:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301786:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301787:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697723.301790:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00100000:1.0:1787697723.301791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126249216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697723.301793:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301798:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301801:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697723.301803:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697723.301806:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697723.301809:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697723.301815:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697723.301820:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697723.301826:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697723.301828:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697723.301829:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697723.301836:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5624 00000100:00080000:0.0:1787697723.301839:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697723.301841:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697723.301842:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301846:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301853:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697723.301856:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697723.301861:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697723.301861:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301866:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301867:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697723.301868:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697723.301872:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697723.301875:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697723.301881:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697723.301886:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697723.301892:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697723.301894:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697723.301895:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697723.301897:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5624 00000100:00080000:0.0:1787697723.301900:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697723.301902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301903:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301905:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697723.301906:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697723.301908:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697723.301910:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697723.301911:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697723.301912:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697723.301915:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697723.301921:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697723.301928:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697723.301935:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697723.301942:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5634) 00000100:00000040:1.0:1787697723.301952:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697723.301971:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697723.301991:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697723.301999:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697723.302014:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787697723.302021:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697723.302030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697723.302040:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697723.302045:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697723.302054:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787697723.302058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697723.302063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697723.302072:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697723.302076:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697723.302083:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787697723.302103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697723.302110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697723.302120:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697723.302124:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697723.302132:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787697723.813572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697723.813572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697723.813576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697723.941580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697724.837483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697724.837487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697724.837487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697724.965604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697725.221512:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300298607, at 4300298624 00000100:00000040:3.0:1787697725.221512:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300298607, at 4300298624 00000100:00100000:0.0:1787697725.861528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697725.861529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697725.861529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697725.989647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697726.373688:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:2.0:1787697726.373721:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697726.373734:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697726.373765:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697726.373782:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697726.373790:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697726.373800:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697726.373809:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697726.373820:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787697726.373831:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697723/real 1787697726] req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697726.373831:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697726.373849:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697726.373856:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697748 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.373883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697726.373897:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697726.373908:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697726.373913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697726.373917:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.373926:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249344:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697726.373933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697723/real 1787697726] req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.373946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697726.373953:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697726.373961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697726.373965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697726.373968:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.373975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249472:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697726.373982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697723/real 1787697726] req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.373999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697726.374007:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697726.374014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697726.374017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697726.374021:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.374028:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249600:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697726.374047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697723/real 1787697726] req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.374058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697726.374066:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697726.374073:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697726.374076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697726.374080:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.374087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249728:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697726.374094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697723/real 1787697726] req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.374105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697726.374112:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697726.374119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697726.374122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697726.374125:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697726.374132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126249216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697726.374137:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126249344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697726.374150:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697726.374159:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126249472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697726.374167:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697726.374172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126249600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697726.374180:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697726.374186:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126249728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697726.374193:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697726.374198:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126249216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697748 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697726.374205:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697726.885503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697726.885503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697726.885510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697727.013586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697727.909501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697727.909501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697727.909512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697728.037589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697728.421556:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697728.421579:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697728.421585:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697728.421597:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5629 00000100:00080000:0.0:1787697728.421611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697728.421615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.421619:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.421623:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697728.421629:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697728.421633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.421636:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.421640:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697728.421643:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697728.421660:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697728.421695:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697728.421736:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697728.421761:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697728.421765:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697728.421767:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697728.421772:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5629 00000100:00080000:0.0:1787697728.421776:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697728.421779:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.421782:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.421784:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697728.421792:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697728.421796:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.421799:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.421801:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697728.421804:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697728.421816:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697728.421822:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697728.421823:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697728.421841:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697728.421845:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126249856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697728.421861:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697728.421871:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697728.421875:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697728.421877:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697728.421890:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5629 00000100:00000040:1.0:1787697728.421891:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697728.421897:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697728.421904:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697728.421909:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697728.421912:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.421916:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697728.421919:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697728.421923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.421926:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.421928:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697728.421931:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697728.421937:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697728.421942:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697728.421953:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697728.421967:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697728.421968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697728.421979:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697728.421984:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697728.421986:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697728.421991:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5629 00000100:00080000:0.0:1787697728.421996:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697728.422013:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697728.422017:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.422026:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.422030:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697728.422033:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697728.422037:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.422047:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.422050:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697728.422053:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697728.422060:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697728.422065:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697728.422068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697728/real 1787697728] req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697728.422079:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697728.422096:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697728.422125:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697728.422129:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697728.422131:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697728.422134:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697728.422143:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5629 00000100:00080000:1.0:1787697728.422149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697728.422150:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697728.422157:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697728.422159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697728.422165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697728.422179:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697728.422181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126249856:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697728.422184:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697728.422189:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697728.422195:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126249856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697728.422196:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697728.422202:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697728.422207:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697728.422213:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697728.422219:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697728.422220:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:0.0:1787697728.422225:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697728.422236:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697728.422242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697728.422269:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697728.422291:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5639) 00000100:00000040:1.0:1787697728.422298:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422309:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697728.422324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697728.422331:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697728.422351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697728/real 1787697728] req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422373:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697728.422381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697728.422384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697728.422390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126249984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697728.422405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422414:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697728.422463:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422471:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697728.422483:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697728.422490:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422500:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697728.422506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697728/real 1787697728] req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697728.422541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697728.422544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697728.422549:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697728.422571:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697728.422598:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422606:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697728.422618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697728.422625:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422637:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697728.422644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697728/real 1787697728] req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422655:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422663:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697728.422671:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697728.422674:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697728.422678:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422687:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697728.422694:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126249984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697728.422701:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697728.422708:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126250112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697728.422716:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697728.422722:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126250240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697728.422729:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697728.422739:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697728.422760:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422766:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697728.422777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697728.422784:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697753 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422794:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697728.422799:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697728/real 1787697728] req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422810:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697728.422819:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697728.422827:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697728.422830:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697728.422834:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697728.422841:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697728.422847:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126250368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697753 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697728.422855:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697728.933545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697728.933546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697728.933555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697729.061606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697729.957507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697729.957507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697729.957507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697730.085563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697730.345477:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300303727, at 4300303748 00000100:00000040:2.0:1787697730.345690:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300303727, at 4300303748 00000100:00100000:1.0:1787697730.981487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697730.981494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697730.981495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697731.109589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697732.005525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697732.005525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697732.005525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697732.133563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697733.029514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697733.029514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697733.029514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697733.157592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697733.541544:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697733.541557:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697733.541562:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697733.541581:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5634 00000100:00080000:0.0:1787697733.541594:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697733.541598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541605:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697733.541609:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697733.541611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541616:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697733.541618:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541633:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697733.541666:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697733.541678:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697733.541697:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697733.541700:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697733.541701:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697733.541704:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5634 00000100:00080000:0.0:1787697733.541726:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697733.541729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541731:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541733:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697733.541735:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697733.541737:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541739:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541742:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787697733.541745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697733.541746:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697733.541757:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697733.541762:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697733.541763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126250496:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787697733.541773:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697733.541780:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697733.541786:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697733.541788:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697733.541789:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697733.541792:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5634 00000100:00000040:1.0:1787697733.541795:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697733.541796:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697733.541803:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541809:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541812:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787697733.541814:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541815:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697733.541820:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541822:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541827:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697733.541830:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697733.541833:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697733.541836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697733.541843:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697733.541849:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697733.541856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697733.541858:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697733.541862:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697733.541863:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697733.541867:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5634 00000100:00080000:0.0:1787697733.541870:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697733.541873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541875:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541877:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697733.541880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697733.541880:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697733.541888:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.541896:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.541899:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697733.541901:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697733.541907:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697733.541937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697733/real 1787697733] req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697733.541940:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697733.541953:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697733.541958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.541972:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697733.541983:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697733.541985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697733.541994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787697733.541995:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697733.541999:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:1.0:1787697733.541999:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697733.542001:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697733.542014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126250496:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697733.542016:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5634 00000100:00000040:1.0:1787697733.542021:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126250496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697733.542023:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697733.542030:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.542033:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.542036:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697733.542036:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:0.0:1787697733.542038:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697733.542042:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697733.542044:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697733.542046:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697733.542048:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697733.542052:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697733.542054:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697733.542057:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697733.542066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250624:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697733.542069:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697733.542077:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542082:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697733.542088:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5644) 00000100:00000040:1.0:1787697733.542091:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697733.542102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697733.542108:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542116:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697733.542120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697733/real 1787697733] req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697733.542137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697733.542139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697733.542141:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697733.542148:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697733.542165:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542172:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697733.542183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697733.542190:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542199:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697733.542205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697733/real 1787697733] req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542223:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697733.542229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697733.542231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697733.542233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697733.542241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697733.542255:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542259:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697733.542266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697733.542270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542279:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697733.542284:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697733/real 1787697733] req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697733.542311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697733.542313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697733.542315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126250880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697733.542324:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126250624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697733.542331:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697733.542337:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126250752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697733.542345:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697733.542349:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126250880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697733.542357:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697733.542364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542373:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697733.542389:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542396:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697733.542408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697733.542415:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697758 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542448:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697733.542455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697733/real 1787697733] req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542467:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697733.542475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697733.542482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697733.542485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697733.542489:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697733.542496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697733.542502:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126251008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697758 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697733.542510:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697734.053523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697734.053529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697734.053571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697734.181610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697735.077567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697735.077567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697735.077567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697735.205598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697735.461511:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300308847, at 4300308864 00000100:00000040:3.0:1787697735.461511:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300308847, at 4300308864 00000100:00100000:1.0:1787697736.101508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697736.101509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697736.101509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697736.229541:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697737.125515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697737.125516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697737.125520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697737.253602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697738.149490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697738.149493:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697738.149506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697738.277575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697738.661539:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697738.661546:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697738.661562:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697738.661569:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5639 00000100:00080000:0.0:1787697738.661577:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697738.661579:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661582:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661585:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697738.661588:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697738.661590:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661592:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661594:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697738.661596:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661604:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697738.661624:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697738.661639:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697738.661659:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697738.661663:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697738.661664:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697738.661669:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5639 00000100:00080000:0.0:1787697738.661672:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697738.661675:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661685:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661687:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697738.661689:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697738.661691:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661693:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661699:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697738.661701:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697738.661708:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697738.661711:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697738.661712:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697738.661724:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697738.661731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126251136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697738.661735:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697738.661744:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697738.661746:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697738.661747:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697738.661750:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5639 00000100:00080000:0.0:1787697738.661753:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697738.661754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661757:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697738.661759:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697738.661761:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697738.661765:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697738.661766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661769:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697738.661772:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697738.661777:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697738.661782:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697738.661787:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697738.661791:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697738.661813:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697738.661833:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787697738.661835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697738.661838:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697738.661840:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697738.661844:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5639 00000100:00080000:0.0:1787697738.661848:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697738.661850:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661852:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661854:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697738.661857:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697738.661857:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697738.661862:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661866:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661870:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697738.661874:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697738.661880:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697738.661883:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697738.661891:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697738.661898:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.661903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697738/real 1787697738] req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697738.661905:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697738.661911:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697738.661913:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697738.661922:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5639 00000100:00080000:0.0:1787697738.661925:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697738.661926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697738.661928:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661930:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697738.661932:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697738.661936:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697738.661936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697738.661941:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697738.661952:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787697738.661953:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697738.661956:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697738.661962:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697738.661968:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697738.661971:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697738.661976:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697738.661979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697738.661988:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697738.661989:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697738.662016:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5649) 00000100:00100000:1.0:1787697738.662019:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126251136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697738.662029:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126251136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697738.662040:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697738.662053:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697738.662075:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662083:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697738.662095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697738.662102:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662113:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697738.662119:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697738/real 1787697738] req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662130:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697738.662146:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697738.662149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697738.662154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697738.662167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697738.662186:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662193:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697738.662203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697738.662208:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662217:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697738.662222:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697738/real 1787697738] req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697738.662249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697738.662252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697738.662256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697738.662282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697738.662302:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662309:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697738.662321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697738.662327:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662336:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697738.662341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697738/real 1787697738] req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697738.662377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697738.662380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697738.662383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697738.662397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697738.662416:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662443:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697738.662455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697738.662462:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697763 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662471:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697738.662476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697738/real 1787697738] req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697738.662495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697738.662502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697738.662505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697738.662509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697738.662516:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697738.662521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126251264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697738.662534:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697738.662540:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126251392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697738.662547:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697738.662553:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126251520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697738.662560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697738.662564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126251648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697763 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697738.662570:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697739.173517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697739.173517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697739.173518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697739.301594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697740.197481:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697740.197491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697740.197509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697740.325607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697740.581494:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300313967, at 4300313984 00000100:00000040:2.0:1787697740.581501:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300313967, at 4300313984 00000100:00100000:1.0:1787697741.221525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697741.221525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697741.221530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697741.349584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697742.245522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697742.245522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697742.245600:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697742.373602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697743.269517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697743.269564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697743.269641:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697743.399513:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697743.781475:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697743.781485:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697743.781488:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697743.781498:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5644 00000100:00080000:0.0:1787697743.781508:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697743.781512:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781516:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781532:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697743.781536:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697743.781538:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781540:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781542:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697743.781544:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781559:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697743.781580:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697743.781590:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697743.781608:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697743.781610:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697743.781611:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697743.781614:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5644 00000100:00080000:0.0:1787697743.781617:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697743.781618:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781620:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781621:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697743.781623:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697743.781625:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781626:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781627:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697743.781629:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697743.781635:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697743.781638:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697743.781654:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697743.781654:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697743.781677:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.781683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126251776:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697743.781696:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697743.781700:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697743.781702:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697743.781706:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5644 00000100:00080000:0.0:1787697743.781711:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697743.781711:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697743.781719:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697743.781727:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781732:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697743.781761:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697743.781766:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781769:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781771:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697743.781775:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:1.0:1787697743.781776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787697743.781782:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697743.781788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697743.781799:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697743.781808:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697743.781813:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697743.781829:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697743.781832:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697743.781839:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697743.781845:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5644 00000100:00100000:1.0:1787697743.781847:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697743/real 1787697743] req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697743.781851:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697743.781857:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781860:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781863:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:1.0:1787697743.781863:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697743.781867:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697743.781873:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781876:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781879:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787697743.781879:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697743.781885:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697743.781893:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697743.781894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697743.781899:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787697743.781900:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697743.781904:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697743.781912:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697743.781912:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126251776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697743.781922:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126251776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697743.781927:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697743.781935:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787697743.781936:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697743.781939:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697743.781941:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697743.781943:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5644 00000100:00080000:0.0:1787697743.781945:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697743.781947:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697743.781952:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697743.781953:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781957:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697743.781959:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787697743.781964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697743.781970:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697743.781972:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697743.781974:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697743.781975:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697743.781979:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697743.781980:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697743.781985:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697743.781989:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697743.782000:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697743.782005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697743.782013:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697743.782018:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782022:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697743.782031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697743/real 1787697743] req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697743.782035:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5654) 00000100:00100000:1.0:1787697743.782041:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782049:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697743.782054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697743.782056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697743.782058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126251904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697743.782065:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782070:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697743.782076:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782080:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697743.782085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697743.782089:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782098:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697743.782101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697743/real 1787697743] req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782107:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782112:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697743.782116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697743.782117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697743.782119:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697743.782126:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697743.782137:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782142:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697743.782146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697743.782150:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782155:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697743.782158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697743/real 1787697743] req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697743.782173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697743.782175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697743.782176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697743.782183:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126251904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697743.782187:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697743.782190:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126252032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697743.782194:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697743.782196:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126252160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697743.782200:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697743.782204:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697743.782215:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782219:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697743.782224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697743.782227:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697768 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782232:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697743.782235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697743/real 1787697743] req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697743.782246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697743.782249:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697743.782251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697743.782253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697743.782256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697743.782259:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126252288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697768 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697743.782263:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697744.293513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697744.293518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697744.293541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697744.421606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697745.317526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697745.317526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697745.317528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697745.445606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697745.701477:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300319087, at 4300319104 00000100:00000040:3.0:1787697745.701485:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300319087, at 4300319104 00000100:00100000:1.0:1787697746.341499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697746.341499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697746.341501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697746.469555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697747.365527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697747.365527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697747.365629:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697747.493589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697748.389506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697748.389508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697748.389546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697748.517620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697748.901494:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697748.901501:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697748.901505:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697748.901515:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5649 00000100:00080000:0.0:1787697748.901525:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697748.901529:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901532:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901536:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697748.901541:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697748.901544:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901547:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901550:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697748.901565:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901573:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697748.901593:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697748.901604:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697748.901622:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697748.901624:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697748.901626:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697748.901629:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5649 00000100:00080000:0.0:1787697748.901631:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697748.901633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901634:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901636:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697748.901638:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697748.901640:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901642:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901643:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697748.901645:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697748.901663:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697748.901667:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697748.901676:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697748.901676:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697748.901702:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.901707:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126252416:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697748.901719:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697748.901721:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697748.901723:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697748.901726:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5649 00000100:00080000:0.0:1787697748.901730:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697748.901732:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697748.901734:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901740:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901744:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697748.901748:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697748.901752:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901760:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901761:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697748.901764:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697748.901767:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697748.901775:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697748.901781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697748.901785:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697748.901793:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.901797:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697748.901800:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697748.901805:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697748.901807:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697748.901811:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5649 00000100:00080000:0.0:1787697748.901817:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697748.901823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901825:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901827:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697748.901829:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697748.901831:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901833:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901835:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697748.901838:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697748.901840:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697748/real 1787697748] req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697748.901842:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697748.901847:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697748.901856:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697748.901865:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.901867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697748.901879:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697748.901885:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697748.901887:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697748.901890:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697748.901892:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5649 00000100:00080000:0.0:1787697748.901897:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697748.901902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697748.901905:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697748.901905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787697748.901910:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:1.0:1787697748.901913:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787697748.901914:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697748.901921:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697748.901922:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697748.901925:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697748.901929:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787697748.901933:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126252416:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697748.901939:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697748.901942:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126252416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697748.901945:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697748.901951:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787697748.901954:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697748.901959:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697748.901967:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.901970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697748.901986:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5660) 00000100:00100000:1.0:1787697748.901987:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697748.902003:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902011:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697748.902024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697748.902031:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902041:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697748.902047:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697748/real 1787697748] req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697748.902075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697748.902078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697748.902083:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697748.902095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697748.902115:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902122:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697748.902133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697748.902140:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902149:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697748.902154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697748/real 1787697748] req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902172:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697748.902180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697748.902183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697748.902187:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697748.902206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697748.902226:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902233:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697748.902244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697748.902250:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902259:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697748.902264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697748/real 1787697748] req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697748.902291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697748.902294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697748.902297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697748.902311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126252544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697748.902319:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697748.902325:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126252672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697748.902331:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697748.902336:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126252800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697748.902343:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697748.902350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697748.902371:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902378:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697748.902390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697748.902396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697773 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902405:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697748.902411:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697748/real 1787697748] req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697748.902460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697748.902467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697748.902471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697748.902475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697748.902482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126252928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697748.902488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126252928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697773 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697748.902496:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697749.413498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697749.413498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697749.413499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697749.541569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697750.437505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697750.437505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697750.437509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697750.565601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697750.821536:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300324207, at 4300324224 00000100:00000040:3.0:1787697750.821536:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300324207, at 4300324224 00000100:00100000:1.0:1787697751.461510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697751.461512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697751.461518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697751.589570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697752.485478:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697752.485479:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697752.485490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697752.613617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697753.509533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697753.509533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697753.509533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697753.637606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697754.021530:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697754.021540:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697754.021545:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697754.021554:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5655 00000100:00080000:0.0:1787697754.021577:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697754.021579:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021583:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021586:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697754.021593:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697754.021596:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021597:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021600:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697754.021602:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021618:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697754.021669:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697754.021686:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697754.021707:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697754.021710:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697754.021711:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697754.021714:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5655 00000100:00080000:0.0:1787697754.021717:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697754.021719:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021721:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021723:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697754.021725:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697754.021727:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021728:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021730:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697754.021733:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697754.021741:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697754.021745:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697754.021751:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697754.021754:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697754.021780:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.021786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126253056:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697754.021794:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697754.021798:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697754.021806:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697754.021812:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5655 00000100:00080000:0.0:1787697754.021816:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697754.021819:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697754.021823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021831:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021836:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787697754.021836:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021841:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697754.021845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021848:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021850:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697754.021853:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697754.021858:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697754.021863:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697754.021872:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697754.021879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021885:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697754.021891:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697754.021894:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697754.021895:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697754.021899:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697754.021900:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5655 00000100:00080000:0.0:1787697754.021905:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697754.021910:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021913:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021914:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697754.021915:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697754.021922:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.021924:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.021926:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697754.021928:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697754.021933:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697754.021936:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697754.021938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697754.021945:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697754.021961:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697754.021967:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697754.021970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697754.021972:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697754.021976:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697754.021985:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5655 00000100:00080000:0.0:1787697754.021992:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697754.021992:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697754.021999:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697754.022006:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697754.022007:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.022014:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697754.022017:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697754.022022:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:1.0:1787697754.022023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126253056:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697754.022030:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697754.022031:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126253056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697754.022040:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697754.022044:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697754.022045:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697754.022050:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697754.022054:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697754.022059:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697754.022063:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697754.022066:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697754.022074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253184:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697754.022080:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697754.022089:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022108:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697754.022112:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697754.022119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697754.022123:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5665) 00000100:00000040:1.0:1787697754.022127:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022136:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697754.022140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022152:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.022157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.022159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697754.022162:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697754.022174:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697754.022195:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022203:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697754.022212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697754.022217:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697754.022227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.022245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.022246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697754.022248:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697754.022256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697754.022268:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022273:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697754.022279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697754.022282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022288:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697754.022291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.022310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.022311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697754.022313:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697754.022320:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126253184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697754.022325:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697754.022329:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126253312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697754.022333:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697754.022336:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126253440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697754.022341:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697754.022345:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697754.022357:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022361:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697754.022366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697754.022370:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697779 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022375:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697754.022378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697754/real 1787697754] req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697754.022390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697754.022393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697754.022395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697754.022397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697754.022401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697754.022404:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126253568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697779 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697754.022408:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697754.533523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697754.533523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697754.533586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697754.661629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697755.557526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697755.557556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697755.557558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697755.685576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697755.941553:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300329327, at 4300329344 00000100:00000040:3.0:1787697755.941553:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300329327, at 4300329344 00000100:00100000:1.0:1787697756.581547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697756.581550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697756.581568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697756.709659:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697757.605518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697757.605528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697757.605563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697757.733666:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697758.629552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697758.629573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697758.629583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697758.757651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697759.141519:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.141536:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697759.141540:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.141551:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5660 00000100:00080000:0.0:1787697759.141563:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697759.141565:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141569:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141572:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697759.141577:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697759.141580:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141582:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141584:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697759.141587:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141600:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697759.141626:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697759.141637:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.141655:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.141675:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697759.141676:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.141680:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5660 00000100:00080000:0.0:1787697759.141683:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697759.141686:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141688:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141689:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697759.141692:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697759.141694:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141697:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141698:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697759.141700:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697759.141709:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697759.141713:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697759.141718:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697759.141723:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697759.141748:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.141756:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697759.141756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126253696:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697759.141761:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697759.141764:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.141768:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5660 00000100:00080000:0.0:1787697759.141770:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697759.141772:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141774:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141780:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697759.141782:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697759.141785:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141788:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141790:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697759.141793:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697759.141797:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697759.141801:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697759.141811:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697759.141816:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697759.141829:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697759.141840:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.141849:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.141852:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697759.141853:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.141857:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5660 00000100:00080000:0.0:1787697759.141860:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697759.141864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141865:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141869:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141871:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697759.141873:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697759.141875:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141877:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141879:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697759.141881:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697759.141885:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697759.141887:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697759.141896:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697759.141909:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697759.141916:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697759.141921:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697759.141923:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697759.141924:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697759.141926:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5660 00000100:00080000:0.0:1787697759.141930:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697759.141934:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697759.141935:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141941:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141944:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697759.141948:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697759.141953:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697759.141956:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697759.141957:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697759.141959:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697759.141963:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697759.141966:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697759.141967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697759.141974:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697759.141989:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697759.141990:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697759.142009:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5670) 00000100:00080000:1.0:1787697759.142013:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.142021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697759.142028:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126253696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697759.142049:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126253696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697759.142060:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697759.142079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697759.142103:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142111:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697759.142124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697759.142131:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142141:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697759.142147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142158:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.142173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.142177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697759.142181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697759.142194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697759.142214:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142231:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697759.142242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697759.142248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142257:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697759.142262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.142297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.142300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697759.142304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126253952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697759.142323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697759.142344:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142351:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697759.142360:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697759.142366:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142386:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697759.142392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142411:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.142438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.142440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697759.142443:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697759.142452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697759.142466:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142470:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697759.142491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697759.142496:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697784 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142502:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697759.142505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697759/real 1787697759] req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697759.142517:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697759.142521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697759.142523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697759.142525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697759.142529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697759.142532:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126253824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697759.142553:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697759.142559:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126253952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697759.142567:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697759.142572:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126254080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697759.142579:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697759.142583:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126254208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697784 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697759.142591:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697759.653486:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697759.653525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697759.653527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697759.781633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697760.677547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697760.677548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697760.677553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697760.805607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697761.061537:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300334447, at 4300334464 00000100:00000040:3.0:1787697761.061541:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300334447, at 4300334464 00000100:00100000:1.0:1787697761.701555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697761.701556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697761.701562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697761.829651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697762.725537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697762.725538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697762.725560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697762.853582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697763.749549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697763.749549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697763.749550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697763.877550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697764.261529:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697764.261539:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697764.261542:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697764.261552:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5665 00000100:00080000:2.0:1787697764.261561:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697764.261565:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.261569:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261572:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697764.261577:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697764.261580:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.261582:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261585:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697764.261588:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261601:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697764.261623:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697764.261638:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697764.261659:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697764.261662:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697764.261664:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697764.261668:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5665 00000100:00080000:2.0:1787697764.261671:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697764.261674:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.261677:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261680:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697764.261683:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697764.261686:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.261689:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261692:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787697764.261696:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697764.261697:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697764.261709:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697764.261712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126254336:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787697764.261717:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697764.261730:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697764.261738:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697764.261744:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.261760:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697764.261762:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697764.261766:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697764.261768:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697764.261773:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5665 00000100:00080000:2.0:1787697764.261776:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697764.261779:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.261781:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261784:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697764.261786:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697764.261789:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.261792:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261795:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697764.261805:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:1.0:1787697764.261809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:2.0:1787697764.261812:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697764.261818:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697764.261827:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697764.261838:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697764.261850:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.261855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697764/real 1787697764] req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697764.261867:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697764.261871:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697764.261874:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697764.261875:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697764.261880:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5665 00000100:00000040:1.0:1787697764.261887:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697764.261887:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697764.261894:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697764.261898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787697764.261899:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697764.261904:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:2.0:1787697764.261904:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697764.261909:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697764.261910:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697764.261914:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.261920:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697764.261923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126254336:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787697764.261925:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697764.261931:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697764.261932:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126254336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697764.261939:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697764.261941:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:2.0:1787697764.261944:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697764.261956:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697764.261957:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.261983:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697764.261987:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.261997:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697764.261998:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697764.262007:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697764.262010:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697764.262013:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697764.262018:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5665 00000100:00080000:2.0:1787697764.262023:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787697764.262026:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697764.262030:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697764.262035:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697764.262036:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.262041:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697764.262046:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697764.262049:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:2.0:1787697764.262050:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697764.262056:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697764.262059:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787697764.262060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697764/real 1787697764] req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697764.262064:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697764.262071:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697764.262077:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697764.262081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697764.262087:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697764.262098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697764.262106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697764.262111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697764.262116:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697764.262124:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697764.262135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697764.262145:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5675) 00000100:00100000:1.0:1787697764.262146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697764.262159:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262164:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697764.262173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697764.262178:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262187:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697764.262190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697764/real 1787697764] req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697764.262206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697764.262208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697764.262210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697764.262217:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126254464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697764.262221:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697764.262225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126254592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697764.262229:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697764.262234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697764.262246:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262251:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697764.262256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697764.262260:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262265:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697764.262268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697764/real 1787697764] req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262279:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697764.262283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697764.262284:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697764.262286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697764.262293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262297:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697764.262303:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262310:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697764.262315:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697764.262318:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697789 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262323:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697764.262326:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697764/real 1787697764] req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697764.262336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697764.262340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697764.262341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697764.262343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697764.262347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126254848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697764.262350:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126254720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697764.262354:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697764.262357:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126254848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697789 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697764.262363:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697764.773525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697764.773525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697764.773560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697764.901545:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697765.797511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697765.797511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697765.797511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697765.925597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697766.181517:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300339567, at 4300339584 00000100:00000040:3.0:1787697766.181517:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300339567, at 4300339584 00000100:00100000:1.0:1787697766.821495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697766.821499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697766.821499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697766.949583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697767.845499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697767.845500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697767.845506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697767.973576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697768.869511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697768.869511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697768.869511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697768.997588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697769.381542:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697769.381549:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697769.381552:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697769.381560:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5670 00000100:00080000:0.0:1787697769.381569:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697769.381571:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.381574:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381576:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697769.381580:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697769.381585:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.381591:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381594:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697769.381596:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381606:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697769.381636:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697769.381651:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697769.381674:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697769.381677:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697769.381679:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697769.381683:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5670 00000100:00080000:0.0:1787697769.381698:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697769.381702:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.381705:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381708:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697769.381711:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697769.381714:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.381717:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381720:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697769.381723:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697769.381730:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697769.381731:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697769.381734:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697769.381747:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697769.381752:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126254976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697769.381760:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697769.381771:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697769.381774:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697769.381776:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697769.381779:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5670 00000100:00080000:0.0:1787697769.381784:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697769.381785:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697769.381791:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.381798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697769.381804:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381814:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697769.381817:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697769.381821:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.381824:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381832:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697769.381835:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:1.0:1787697769.381840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787697769.381844:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697769.381850:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697769.381858:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697769.381861:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697769.381884:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697769.381893:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697769.381896:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697769.381898:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697769.381905:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5670 00000100:00100000:1.0:1787697769.381906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697769/real 1787697769] req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697769.381911:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697769.381918:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.381922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381924:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697769.381926:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697769.381928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697769.381932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697769.381943:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.381948:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787697769.381949:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697769.381955:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:1.0:1787697769.381964:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787697769.381965:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697769.381971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787697769.381973:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697769.381977:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697769.381987:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697769.381989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126254976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697769.382001:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126254976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697769.382012:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697769.382024:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382032:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382064:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255104:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787697769.382065:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697769.382072:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697769.382074:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697769.382078:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5670 00000100:00000040:1.0:1787697769.382083:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697769.382085:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697769.382092:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697769.382099:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697769.382100:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697769.382104:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697769.382107:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697769.382110:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697769.382113:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697769.382118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697769.382119:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697769.382125:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697769.382130:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697769.382132:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697769.382141:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697769.382147:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787697769.382155:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697769.382158:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697769/real 1787697769] req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697769.382184:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697769.382209:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5680) 00000100:00000040:1.0:1787697769.382214:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697769.382223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697769.382227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697769.382233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697769.382246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697769.382266:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382273:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697769.382285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697769.382291:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382315:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697769.382321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697769/real 1787697769] req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382332:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382340:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697769.382348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697769.382350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697769.382354:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697769.382371:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126255104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697769.382379:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697769.382385:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126255232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697769.382392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697769.382399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697769.382448:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382456:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697769.382468:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697769.382475:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382483:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697769.382489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697769/real 1787697769] req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382507:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697769.382515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697769.382518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697769.382521:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697769.382534:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697769.382553:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382560:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697769.382570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697769.382577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697794 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382586:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697769.382591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697769/real 1787697769] req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697769.382609:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697769.382615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697769.382619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697769.382622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697769.382629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697769.382635:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126255360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697769.382642:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697769.382648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126255488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697794 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697769.382654:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697769.893490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697769.893491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697769.893502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697770.021579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697770.917522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697770.917522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697770.917536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697771.045572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697771.301531:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300344687, at 4300344704 00000100:00000040:2.0:1787697771.301534:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300344687, at 4300344704 00000100:00100000:1.0:1787697771.941520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697771.941534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697771.941583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697772.069594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697772.965530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697772.965530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697772.965554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697773.093553:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697773.989515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697773.989515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697773.989612:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697774.117587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697774.501527:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697774.501533:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697774.501536:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697774.501543:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5675 00000100:00080000:3.0:1787697774.501551:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697774.501553:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501556:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501558:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697774.501562:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697774.501564:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501565:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501568:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697774.501570:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501580:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697774.501599:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697774.501609:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697774.501625:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697774.501628:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697774.501629:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697774.501631:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5675 00000100:00080000:3.0:1787697774.501644:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697774.501647:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501649:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501651:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697774.501653:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697774.501655:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501657:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501658:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697774.501660:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697774.501666:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697774.501667:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697774.501673:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697774.501683:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697774.501691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126255616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697774.501697:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697774.501707:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697774.501709:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697774.501711:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697774.501715:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5675 00000100:00000040:1.0:1787697774.501716:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697774.501720:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697774.501725:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501730:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501735:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697774.501738:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697774.501744:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697774.501747:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501749:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501757:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697774.501759:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697774.501763:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697774.501766:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697774.501774:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697774.501780:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501787:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697774.501793:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697774.501795:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697774.501797:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697774.501802:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697774.501803:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5675 00000100:00080000:3.0:1787697774.501809:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697774.501815:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501820:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501822:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697774.501824:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697774.501835:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697774.501842:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697774/real 1787697774] req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697774.501843:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501847:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697774.501852:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697774.501859:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697774.501864:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697774.501870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697774.501874:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697774.501902:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697774.501925:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787697774.501928:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697774.501933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787697774.501939:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697774.501943:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697774.501944:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697774.501948:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697774.501955:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5675 00000100:00100000:1.0:1787697774.501962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126255616:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697774.501963:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697774.501969:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501973:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501975:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697774.501978:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697774.501979:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126255616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697774.501981:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697774.501986:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697774.501990:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697774.501995:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697774.502001:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697774.502005:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:3.0:1787697774.502005:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697774.502014:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697774.502029:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697774.502043:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5685) 00000100:00100000:1.0:1787697774.502049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697774.502065:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502077:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697774.502089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697774.502096:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502106:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697774.502112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697774/real 1787697774] req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697774.502137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697774.502140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697774.502144:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502151:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697774.502156:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697774.502174:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502181:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697774.502191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697774.502198:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502206:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697774.502224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697774/real 1787697774] req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697774.502253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697774.502256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697774.502260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126255872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697774.502273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697774.502294:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502299:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697774.502307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697774.502311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502320:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697774.502324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697774/real 1787697774] req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697774.502344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697774.502347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697774.502351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697774.502364:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126255744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697774.502372:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697774.502378:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126255872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697774.502385:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697774.502389:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126256000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697774.502393:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697774.502398:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697774.502412:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502439:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697774.502448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697774.502453:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697799 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502458:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697774.502462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697774/real 1787697774] req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697774.502473:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697774.502477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697774.502479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697774.502481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697774.502484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697774.502488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126256128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697799 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697774.502492:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697775.013500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697775.013500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697775.013500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697775.141583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697776.037504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697776.037504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697776.037505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697776.165598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697776.421534:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300349807, at 4300349824 00000100:00000040:3.0:1787697776.421638:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300349807, at 4300349824 00000100:00100000:0.0:1787697777.061497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697777.061497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697777.061498:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697777.189580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697778.085522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697778.085522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697778.085522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697778.213575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697779.109502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697779.109502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697779.109502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697779.237623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697779.621544:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697779.621554:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697779.621558:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697779.621569:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5680 00000100:00080000:0.0:1787697779.621580:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697779.621584:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621589:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621592:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697779.621595:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697779.621598:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621600:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621602:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697779.621604:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621615:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697779.621638:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697779.621653:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697779.621672:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697779.621674:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697779.621676:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697779.621678:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5680 00000100:00080000:0.0:1787697779.621694:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697779.621696:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621698:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621700:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697779.621702:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697779.621704:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621705:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621707:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697779.621712:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697779.621712:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697779.621722:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697779.621727:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697779.621728:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126256256:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787697779.621739:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697779.621747:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697779.621753:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697779.621755:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697779.621756:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697779.621759:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697779.621767:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5680 00000100:00080000:0.0:1787697779.621770:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697779.621772:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621775:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621778:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621779:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697779.621780:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697779.621782:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621785:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621792:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697779.621795:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697779.621800:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697779.621804:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697779.621809:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697779.621818:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697779.621828:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697779.621832:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697779.621853:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697779.621856:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697779.621858:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697779.621863:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5680 00000100:00080000:0.0:1787697779.621868:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697779.621872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697779/real 1787697779] req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697779.621875:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621883:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.621888:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697779.621892:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697779.621895:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697779.621901:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697779.621902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697779.621904:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697779.621910:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697779.621921:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697779.621923:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697779.621928:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697779.621937:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697779.621943:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697779.621944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697779.621954:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.621973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126256256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697779.621982:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.621986:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126256256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697779.622011:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:1.0:1787697779.622013:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787697779.622016:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697779.622019:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697779.622023:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5680 00000100:00080000:0.0:1787697779.622027:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697779.622030:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697779.622034:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697779.622037:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.622042:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697779.622047:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697779.622052:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697779.622055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697779.622059:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697779.622064:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697779.622067:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697779.622073:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697779.622076:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697779.622079:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697779.622090:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697779.622097:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697779.622105:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697779.622114:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5690) 00000800:00000100:1.0:1787697779.622119:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697779.622156:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622165:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697779.622169:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697779/real 1787697779] req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697779.622186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697779.622189:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697779.622191:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697779.622199:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622203:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697779.622211:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622215:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697779.622225:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697779.622231:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622240:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697779.622245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697779/real 1787697779] req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622264:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697779.622272:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697779.622275:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697779.622279:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697779.622292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697779.622313:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622321:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697779.622331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697779.622336:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622342:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697779.622346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697779/real 1787697779] req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697779.622362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697779.622364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697779.622366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697779.622374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126256384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697779.622378:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697779.622382:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126256512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697779.622386:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697779.622389:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126256640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697779.622393:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697779.622397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697779.622408:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622412:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697779.622448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697779.622453:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697804 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622459:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697779.622462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697779/real 1787697779] req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697779.622473:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697779.622478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697779.622480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697779.622482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697779.622486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126256768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697779.622489:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126256768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697804 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697779.622493:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697780.133515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697780.133515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697780.133516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697780.261562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697781.157471:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697781.157490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697781.157493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697781.285598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697781.541488:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300354927, at 4300354944 00000100:00000040:3.0:1787697781.541505:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300354927, at 4300354944 00000100:00100000:1.0:1787697782.181519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697782.181519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697782.181523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697782.309546:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697783.205565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697783.205567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697783.205573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697783.333661:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697784.229571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697784.229571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697784.229573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697784.357604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697784.741575:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697784.741583:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697784.741586:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697784.741593:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5685 00000100:00080000:3.0:1787697784.741602:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697784.741604:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.741607:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697784.741610:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697784.741613:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697784.741617:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.741619:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697784.741622:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697784.741625:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697784.741639:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697784.741668:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697784.741683:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.741707:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697784.741711:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697784.741713:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697784.741717:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5685 00000100:00080000:3.0:1787697784.741720:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697784.741723:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.741728:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697784.741730:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697784.741748:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697784.741750:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.741752:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697784.741754:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697784.741756:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697784.741763:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697784.741770:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697784.741770:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697784.741792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126256896:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787697784.741792:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697784.741813:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.741824:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697784.741827:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697784.741830:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697784.741832:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.741844:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5685 00000100:00080000:3.0:1787697784.741851:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697784.741852:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697784.741861:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.741866:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697784.741868:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697784.741871:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697784.741874:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.741878:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697784.741880:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697784.741882:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697784.741889:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697784.741895:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697784.741899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697784.741919:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697784.741919:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697784.741963:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.741974:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697784.741978:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787697784.741980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697784/real 1787697784] req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.741982:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697784.741997:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5685 00000100:00080000:3.0:1787697784.742008:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697784.742009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697784.742014:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.742024:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697784.742051:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697784.742060:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697784.742064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697784.742071:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697784.742079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787697784.742080:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697784.742084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:3.0:1787697784.742089:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787697784.742090:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.742101:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697784.742106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126256896:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787697784.742118:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697784.742119:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126256896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697784.742132:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787697784.742139:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697784.742149:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697784.742161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697784.742166:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257024:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697784.742199:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697784.742203:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:1.0:1787697784.742207:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.742216:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697784.742231:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5685 00000100:00000040:1.0:1787697784.742232:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697784.742239:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697784.742246:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697784.742249:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697784.742256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697784.742258:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787697784.742267:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697784.742267:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697784.742283:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697784.742287:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787697784.742293:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697784.742297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697784/real 1787697784] req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697784.742303:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787697784.742319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697784.742319:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697784.742337:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697784.742338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697784.742353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787697784.742354:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697784.742358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697784.742366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697784.742380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697784.742382:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697784.742400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697784.742452:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742461:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697784.742473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697784.742482:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697784.742487:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742510:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787697784.742517:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5695) 00000100:00100000:1.0:1787697784.742519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697784/real 1787697784] req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697784.742554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697784.742558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697784.742564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697784.742587:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126257024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697784.742594:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697784.742599:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126257152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697784.742604:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697784.742613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742623:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697784.742637:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742650:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697784.742659:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697784.742664:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742670:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697784.742674:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697784/real 1787697784] req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742686:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697784.742691:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697784.742693:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697784.742695:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697784.742708:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697784.742727:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742735:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697784.742746:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697784.742751:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697809 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742757:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697784.742761:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697784/real 1787697784] req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742767:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697784.742773:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697784.742777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697784.742779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697784.742781:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697784.742785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697784.742788:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126257280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697784.742793:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697784.742797:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126257408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697809 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697784.742801:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697785.253516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697785.253516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697785.253516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697785.381581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697786.277528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697786.277529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697786.277567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697786.405584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697786.661486:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300360047, at 4300360064 00000100:00000040:2.0:1787697786.661489:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300360047, at 4300360064 00000100:00100000:0.0:1787697787.301515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697787.301516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697787.301528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697787.429587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697788.325527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697788.325527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697788.325535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697788.453542:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697789.349506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697789.349506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697789.349506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697789.477589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697789.861541:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697789.861550:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697789.861554:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697789.861564:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5690 00000100:00080000:2.0:1787697789.861574:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697789.861577:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861580:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861584:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697789.861588:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697789.861591:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861593:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861595:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697789.861597:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861613:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697789.861647:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697789.861660:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697789.861684:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697789.861687:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697789.861688:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697789.861691:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5690 00000100:00080000:2.0:1787697789.861694:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697789.861696:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861697:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861699:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697789.861701:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697789.861703:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861704:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861706:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697789.861707:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697789.861716:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697789.861720:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697789.861728:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697789.861734:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697789.861741:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697789.861742:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697789.861748:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697789.861750:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697789.861757:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5690 00000100:00080000:2.0:1787697789.861760:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697789.861763:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697789.861766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126257536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697789.861767:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861771:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697789.861773:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697789.861775:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861776:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861778:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697789.861779:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697789.861782:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697789.861784:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697789.861791:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697789.861796:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697789.861800:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697789.861806:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697789.861812:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697789.861814:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697789.861815:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697789.861818:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5690 00000100:00080000:2.0:1787697789.861821:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697789.861823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861824:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861825:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697789.861827:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697789.861828:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861830:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861832:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697789.861834:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697789.861838:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697789.861841:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697789.861847:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697789.861853:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697789.861858:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697789.861860:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697789.861861:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697789.861863:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5690 00000100:00080000:2.0:1787697789.861865:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697789.861867:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861868:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861870:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697789.861871:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697789.861874:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697789.861876:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697789.861877:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697789.861879:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697789.861882:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697789.861886:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697789.861893:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697789.861893:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697789.861920:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697789.861923:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697789.861936:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5700) 00000100:00000040:1.0:1787697789.861942:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697789.861961:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697789.861974:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787697789.861979:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697789.861985:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697789.861994:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697789.861998:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697789.862006:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787697789.862011:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697789.862020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697789.862033:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697789.862041:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697789.862052:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787697789.862061:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697789.862067:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697789.862075:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697789.862080:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697789.862089:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787697790.373538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697790.373556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697790.373556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697790.501576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697791.397534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697791.397534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697791.397595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697791.525605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697791.781525:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300365167, at 4300365184 00000100:00000040:3.0:1787697791.781525:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300365167, at 4300365184 00000100:00100000:0.0:1787697792.421538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697792.421538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697792.421538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697792.549572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787697792.933639:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787697792.933689:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787697792.933705:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697792.933750:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697792.933768:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697792.933777:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697792.933787:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697792.933795:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697792.933804:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697792.933817:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787697792.933825:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787697792.933832:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697814 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.933929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697789/real 1787697792] req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.933949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697792.933964:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697792.933975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697792.933981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697792.933985:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.933994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258048:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697792.934001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697789/real 1787697792] req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934014:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697792.934022:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697792.934030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697792.934033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697792.934037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257664:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697792.934050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697789/real 1787697792] req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697792.934074:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697792.934082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697792.934085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697792.934089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257792:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697792.934100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697789/real 1787697792] req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697792.934117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697792.934122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697792.934125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697792.934129:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126257920:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697792.934144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697789/real 1787697792] req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934155:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697792.934163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697792.934169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697792.934172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697792.934176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697792.934183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126257536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697792.934189:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126258048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697792.934201:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697792.934211:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126257664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697792.934218:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697792.934223:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126257792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697792.934230:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697792.934235:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126257920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697792.934241:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697792.934247:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126257536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697814 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697792.934253:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697793.445504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697793.445504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697793.445570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697793.573607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697794.469507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697794.469507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697794.469507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697794.597533:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697794.981538:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697794.981545:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697794.981547:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697794.981555:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5695 00000100:00080000:0.0:1787697794.981563:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697794.981566:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981570:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981576:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697794.981580:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697794.981583:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981584:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981586:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697794.981589:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981605:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697794.981625:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697794.981635:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697794.981672:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697794.981675:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697794.981677:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697794.981681:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5695 00000100:00080000:0.0:1787697794.981685:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697794.981688:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981691:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981693:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697794.981696:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697794.981700:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981703:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981706:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697794.981708:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697794.981718:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697794.981719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697794.981725:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697794.981740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126258176:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787697794.981741:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697794.981757:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.981775:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697794.981779:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697794.981787:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697794.981790:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697794.981795:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697794.981797:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5695 00000100:00080000:0.0:1787697794.981802:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697794.981812:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981815:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981817:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697794.981820:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697794.981823:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981826:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981828:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697794.981830:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697794.981835:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697794.981838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981841:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697794.981854:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697794.981877:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.981882:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697794.981893:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697794.981898:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697794.981900:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697794.981906:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5695 00000100:00080000:0.0:1787697794.981910:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697794.981919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981924:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:1.0:1787697794.981926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697794/real 1787697794] req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697794.981927:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697794.981932:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.981937:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.981942:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697794.981953:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697794.981955:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697794.981961:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697794.981970:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697794.981972:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697794.981984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697794.981985:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697794.981991:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697794.982000:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697794.982006:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697794.982025:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697794.982025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126258176:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697794.982034:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697794.982036:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697794.982040:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126258176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697794.982042:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5695 00000100:00080000:0.0:1787697794.982049:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697794.982054:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.982057:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697794.982057:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:0.0:1787697794.982064:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697794.982066:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697794.982070:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697794.982073:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697794.982075:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697794.982080:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697794.982081:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697794.982084:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697794.982092:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697794.982099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258304:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697794.982106:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697794.982123:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697794.982124:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982136:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697794.982148:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5706) 00000800:00000100:1.0:1787697794.982156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697794.982178:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982189:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697794.982195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697794/real 1787697794] req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982214:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697794.982220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697794.982223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697794.982227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982233:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697794.982237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697794.982256:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982262:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697794.982273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697794.982280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982288:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697794.982295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697794/real 1787697794] req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697794.982320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697794.982323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697794.982326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697794.982344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982352:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697794.982369:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982375:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697794.982386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697794.982393:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982401:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697794.982406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697794/real 1787697794] req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697794.982467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697794.982470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697794.982474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697794.982487:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126258304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697794.982495:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697794.982501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126258432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697794.982508:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697794.982513:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126258560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697794.982519:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697794.982527:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982535:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697794.982547:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982554:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697794.982566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697794.982573:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697819 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982582:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697794.982588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697794/real 1787697794] req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697794.982607:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697794.982614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697794.982618:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697794.982621:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697794.982627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697794.982633:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126258688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697819 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697794.982646:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697795.493492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697795.493495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697795.493571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697795.621564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697796.517537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697796.517537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697796.517538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697796.645601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697796.901555:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300370287, at 4300370304 00000100:00000040:3.0:1787697796.902228:0:88332:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300370287, at 4300370304 00000100:00100000:1.0:1787697797.541580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697797.541580:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697797.541580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697797.669608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697798.565492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697798.565509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697798.565514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697798.693546:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697799.589577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697799.589580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697799.589580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697799.717574:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697800.101490:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697800.101500:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697800.101503:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697800.101514:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5701 00000100:00080000:0.0:1787697800.101524:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697800.101527:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101530:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101533:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697800.101538:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697800.101542:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101544:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101552:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697800.101556:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101574:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697800.101604:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697800.101620:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697800.101643:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697800.101645:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697800.101647:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697800.101650:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5701 00000100:00080000:0.0:1787697800.101653:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697800.101655:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101673:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101675:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697800.101677:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697800.101681:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101684:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697800.101684:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697800.101688:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697800.101692:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697800.101703:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787697800.101705:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126258816:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697800.101709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697800.101719:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697800.101726:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697800.101732:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697800.101734:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697800.101736:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697800.101738:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.101750:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697800.101760:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5701 00000100:00080000:0.0:1787697800.101802:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697800.101807:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101812:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697800.101814:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697800.101820:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697800.101821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101828:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697800.101831:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697800.101842:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697800.101843:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697800.101850:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697800.101868:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697800.101876:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.101883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697800.101886:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697800.101892:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697800.101895:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697800.101898:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697800.101901:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5701 00000100:00080000:0.0:1787697800.101909:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697800.101914:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697800.101916:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101923:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101927:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697800.101932:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:1.0:1787697800.101933:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787697800.101936:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697800.101941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697800.101943:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.101947:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697800.101951:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697800.101955:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697800.101957:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697800.101965:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697800.101968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126258816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697800.101978:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126258816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697800.101979:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787697800.101994:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697800.101999:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102018:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697800.102023:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697800.102030:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787697800.102033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258944:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787697800.102034:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697800.102043:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5701 00000100:00080000:0.0:1787697800.102049:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697800.102052:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697800.102056:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697800.102058:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.102063:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697800.102067:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697800.102073:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697800.102074:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697800.102079:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697800.102082:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697800.102085:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697800.102090:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697800.102090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697800.102096:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697800.102098:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697800.102106:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697800.102108:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697800.102117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697800.102120:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697800.102145:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5711) 00000100:00000040:1.0:1787697800.102164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.102171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.102173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697800.102176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126258944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697800.102184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697800.102198:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102203:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697800.102211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697800.102215:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102221:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697800.102225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.102242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.102243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697800.102245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697800.102252:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126258944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697800.102256:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697800.102259:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126259072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697800.102264:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697800.102269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697800.102281:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102289:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697800.102294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697800.102298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102303:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697800.102306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102318:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.102322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.102324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697800.102326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697800.102332:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697800.102342:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102346:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697800.102351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697800.102354:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697825 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102359:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697800.102362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697800/real 1787697800] req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697800.102372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697800.102377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697800.102379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697800.102381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697800.102385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697800.102388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126259200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697800.102391:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697800.102394:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126259328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697825 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697800.102397:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697800.613544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697800.613552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697800.613580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697800.741606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697801.637542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697801.637542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697801.637666:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697801.765591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697802.021539:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300375407, at 4300375424 00000100:00000040:3.0:1787697802.021539:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300375407, at 4300375424 00000100:00100000:0.0:1787697802.661538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697802.661538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697802.661538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697802.789634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697803.685574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697803.685574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697803.685576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697803.813599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697804.709510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697804.709510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697804.709510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697804.837575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697805.221539:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697805.221546:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697805.221563:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697805.221573:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5706 00000100:00080000:3.0:1787697805.221583:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697805.221587:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697805.221591:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221595:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697805.221599:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697805.221603:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697805.221606:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221608:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697805.221611:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221620:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697805.221647:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697805.221662:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697805.221682:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697805.221695:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697805.221697:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697805.221702:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5706 00000100:00080000:3.0:1787697805.221706:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697805.221709:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697805.221711:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221714:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697805.221717:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697805.221720:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697805.221724:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697805.221735:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697805.221736:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697805.221744:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697805.221751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126259456:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787697805.221755:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697805.221759:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697805.221768:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697805.221776:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697805.221779:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.221788:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697805.221797:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697805.221801:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697805.221804:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697805.221808:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5706 00000100:00080000:3.0:1787697805.221813:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697805.221816:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697805.221818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697805.221822:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221826:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697805.221829:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697805.221833:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697805.221835:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697805.221841:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221845:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697805.221849:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697805.221854:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697805.221859:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697805.221871:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697805.221875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697805/real 1787697805] req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.221896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.221912:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697805.221916:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697805.221923:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697805.221930:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697805.221932:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787697805.221935:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697805.221938:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697805.221941:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697805.221945:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5706 00000100:00080000:3.0:1787697805.221951:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697805.221955:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697805.221959:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221963:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697805.221967:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697805.221972:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697805.221973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126259456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697805.221974:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.221978:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697805.221981:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697805.221986:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697805.221987:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126259456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697805.221990:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697805.222000:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787697805.222002:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787697805.222010:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697805.222015:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697805.222017:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697805.222018:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697805.222022:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697805.222026:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5706 00000100:00080000:3.0:1787697805.222033:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697805.222039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787697805.222039:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697805.222046:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697805.222048:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697805.222051:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697805.222054:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697805.222057:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697805.222059:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697805.222061:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697805.222068:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697805.222076:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697805.222077:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697805.222084:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:1.0:1787697805.222093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697805.222097:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697805.222104:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697805.222114:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222120:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787697805.222130:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5716) 00000100:00100000:1.0:1787697805.222132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697805/real 1787697805] req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697805.222165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697805.222168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697805.222172:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697805.222183:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697805.222202:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222209:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697805.222221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697805.222228:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222236:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697805.222242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697805/real 1787697805] req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697805.222267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697805.222270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697805.222274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697805.222293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697805.222312:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222320:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697805.222331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697805.222337:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222344:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697805.222348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697805/real 1787697805] req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697805.222364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697805.222366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697805.222368:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697805.222376:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126259584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697805.222381:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697805.222384:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126259712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697805.222388:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697805.222391:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126259840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697805.222395:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697805.222399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697805.222411:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222415:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697805.222442:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697805.222446:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697830 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222451:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697805.222455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697805/real 1787697805] req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697805.222466:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697805.222470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697805.222472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697805.222474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697805.222478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126259968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697805.222481:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126259968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697830 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697805.222485:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697805.733524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697805.733525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697805.733525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697805.861593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697806.757568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697806.757568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697806.757568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697806.885592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697807.141549:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300380527, at 4300380544 00000100:00000040:2.0:1787697807.141553:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300380527, at 4300380544 00000100:00100000:0.0:1787697807.781517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697807.781518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697807.781545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697807.909618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697808.805525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697808.805525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697808.805525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697808.933565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697809.829522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697809.829522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697809.829522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697809.957532:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697810.341561:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697810.341576:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697810.341581:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697810.341595:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5711 00000100:00080000:0.0:1787697810.341623:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697810.341627:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341630:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341633:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697810.341638:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697810.341641:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341643:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341645:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697810.341648:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341664:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697810.341689:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697810.341701:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697810.341719:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697810.341722:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697810.341724:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697810.341727:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5711 00000100:00080000:0.0:1787697810.341744:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697810.341746:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341748:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341750:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697810.341752:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697810.341754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341755:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341758:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697810.341760:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697810.341769:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697810.341773:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697810.341781:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697810.341788:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697810.341794:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697810.341797:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697810.341798:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697810.341800:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5711 00000100:00080000:0.0:1787697810.341802:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697810.341804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341808:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697810.341809:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697810.341811:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341812:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341817:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697810.341824:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697810.341827:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697810.341830:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697810.341835:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697810.341838:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697810.341871:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.341882:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126260096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697810.341884:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697810.341890:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697810.341892:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697810.341898:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5711 00000100:00080000:0.0:1787697810.341903:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697810.341907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341909:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341912:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:1.0:1787697810.341915:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697810.341917:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697810.341931:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697810.341941:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.341946:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787697810.341951:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697810.341957:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697810.341966:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697810.341973:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697810.341984:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697810.341992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342008:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697810.342014:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697810.342027:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697810.342031:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697810.342034:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697810.342041:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5711 00000100:00100000:1.0:1787697810.342042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697810/real 1787697810] req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697810.342053:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697810.342061:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697810.342061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697810.342075:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697810.342084:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697810.342094:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:1.0:1787697810.342095:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697810.342111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697810.342113:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697810.342127:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697810.342137:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787697810.342142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126260096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697810.342147:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697810.342157:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697810.342161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260224:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787697810.342169:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697810.342177:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697810.342188:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697810.342188:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697810.342205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697810.342212:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342214:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342246:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697810.342256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697810/real 1787697810] req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697810.342257:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5721) 00000100:00100000:1.0:1787697810.342271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342278:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697810.342284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697810.342286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697810.342288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697810.342296:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126260096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697810.342303:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342309:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126260224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697810.342313:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697810.342323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697810.342335:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342340:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697810.342345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342349:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342355:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697810.342359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697810/real 1787697810] req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697810.342386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697810.342389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697810.342392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697810.342404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342411:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697810.342465:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342472:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697810.342482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342501:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697810.342506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697810/real 1787697810] req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342523:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697810.342530:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697810.342533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697810.342536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342544:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697810.342549:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697810.342563:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342567:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697810.342575:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697810.342579:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697835 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342584:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697810.342588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697810/real 1787697810] req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697810.342613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697810.342618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697810.342620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697810.342622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697810.342626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697810.342630:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126260352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697810.342634:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697810.342638:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126260480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697810.342642:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697810.342645:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126260608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697835 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697810.342649:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697810.853495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697810.853496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697810.853507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697810.981614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697811.877506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697811.877529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697811.877539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697812.005516:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697812.261502:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300385647, at 4300385664 00000100:00000040:2.0:1787697812.261513:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300385647, at 4300385664 00000100:00100000:1.0:1787697812.901516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697812.901531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697812.901540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697813.029625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697813.925498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697813.925556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697813.925558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697814.053548:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697814.949515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697814.949515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697814.949544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697815.077604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697815.461500:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697815.461508:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697815.461519:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697815.461527:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5716 00000100:00080000:1.0:1787697815.461537:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697815.461539:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461542:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461544:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697815.461548:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697815.461550:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461551:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461553:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697815.461555:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461566:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697815.461584:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697815.461594:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697815.461611:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697815.461614:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697815.461615:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697815.461617:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5716 00000100:00080000:1.0:1787697815.461620:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697815.461622:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461623:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461625:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697815.461626:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697815.461629:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461630:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461632:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697815.461634:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697815.461641:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697815.461644:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697815.461650:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697815.461652:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697815.461658:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697815.461667:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697815.461672:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697815.461674:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697815.461679:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5716 00000100:00080000:1.0:1787697815.461683:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697815.461684:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461686:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461687:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697815.461689:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00100000:2.0:1787697815.461690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126260736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697815.461693:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461698:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461700:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697815.461702:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697815.461704:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697815.461707:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697815.461713:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697815.461718:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.461722:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697815.461726:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697815.461732:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697815.461736:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697815.461740:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787697815.461744:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5716 00000100:00080000:1.0:1787697815.461746:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697815.461748:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461750:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461751:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697815.461753:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697815.461755:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461757:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697815.461759:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697815.461762:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697815.461765:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697815.461770:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697815.461777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461786:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697815.461791:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697815.461793:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697815.461794:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697815.461798:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5716 00000100:00000040:2.0:1787697815.461799:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697815.461800:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697815.461806:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461811:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461815:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697815.461817:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697815.461819:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697815.461822:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697815.461823:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697815.461824:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697815.461827:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697815.461830:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697815.461836:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697815.461839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697815/real 1787697815] req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697815.461855:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.461861:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697815.461862:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5726) 00000100:00000040:2.0:1787697815.461871:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697815.461878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697815.461880:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697815.461883:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.461888:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126260736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697815.461893:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126260736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697815.461899:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697815.461910:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.461916:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697815.461925:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.461930:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697815.461937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697815.461941:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.461948:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697815.461951:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697815/real 1787697815] req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.461958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.461963:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697815.461967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697815.461968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697815.461971:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.461975:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697815.461980:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.461986:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697815.461995:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462002:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697815.462012:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697815.462019:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462028:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697815.462034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697815/real 1787697815] req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462053:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697815.462061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697815.462064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697815.462066:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126260992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697815.462074:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697815.462088:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462092:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697815.462099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697815.462103:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462109:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697815.462112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697815/real 1787697815] req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462123:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462127:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697815.462132:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697815.462133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697815.462135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697815.462143:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462147:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697815.462153:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462157:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697815.462162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697815.462165:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697840 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462170:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697815.462173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697815/real 1787697815] req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462179:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697815.462184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697815.462188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697815.462190:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697815.462191:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697815.462195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697815.462198:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126260864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697815.462202:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697815.462206:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126260992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697815.462210:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697815.462212:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126261120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697815.462216:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697815.462218:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126261248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697840 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697815.462222:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697815.973521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697815.973521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697815.973579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697816.101616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697816.997523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697816.997523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697816.997527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697817.125620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697817.381519:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300390767, at 4300390784 00000100:00000040:3.0:1787697817.381519:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300390767, at 4300390784 00000100:00100000:0.0:1787697818.021526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697818.021526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697818.021526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697818.149577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697819.045524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697819.045524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697819.045524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697819.173576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697820.069503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697820.069503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697820.069506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697820.197715:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697820.581522:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697820.581530:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697820.581533:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697820.581540:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5721 00000100:00080000:0.0:1787697820.581548:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697820.581550:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581553:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581555:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697820.581559:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697820.581562:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581564:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581566:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697820.581568:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581579:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697820.581599:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697820.581609:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697820.581623:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697820.581626:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697820.581627:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697820.581642:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5721 00000100:00080000:0.0:1787697820.581645:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697820.581647:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581650:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581653:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697820.581656:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697820.581660:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581665:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697820.581667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697820.581668:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697820.581674:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697820.581687:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697820.581694:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697820.581697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126261376:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787697820.581708:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697820.581730:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.581731:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697820.581749:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697820.581753:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787697820.581755:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697820.581759:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697820.581766:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5721 00000100:00080000:0.0:1787697820.581769:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697820.581771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581774:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697820.581776:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697820.581778:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581786:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581790:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000800:00000100:2.0:1787697820.581791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697820.581794:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697820.581800:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697820.581804:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697820.581811:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697820.581817:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697820.581841:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.581844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697820/real 1787697820] req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697820.581857:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697820.581860:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697820.581862:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697820.581865:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5721 00000100:00080000:0.0:1787697820.581869:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697820.581869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697820.581873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581878:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581882:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697820.581885:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697820.581889:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697820.581893:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697820.581894:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581898:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697820.581902:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697820.581910:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697820.581913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787697820.581915:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787697820.581919:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787697820.581924:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697820.581927:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.581956:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126261376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697820.581960:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697820.581968:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697820.581968:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126261376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697820.581974:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697820.581977:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697820.581983:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5721 00000100:00080000:0.0:1787697820.581987:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000020:00000040:2.0:1787697820.581987:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581992:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.581994:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.581996:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697820.581997:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697820.581999:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697820.582000:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697820.582002:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697820.582006:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697820.582007:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697820.582009:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697820.582014:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697820.582022:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697820.582034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697820.582036:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697820.582047:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5731) 00000100:00000040:2.0:1787697820.582054:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582062:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697820.582072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697820.582078:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582087:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697820.582092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697820/real 1787697820] req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697820.582117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697820.582120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697820.582124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697820.582136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697820.582154:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582160:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697820.582169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697820.582174:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582181:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697820.582186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697820/real 1787697820] req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582204:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697820.582211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697820.582214:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697820.582217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697820.582229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697820.582246:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582253:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697820.582262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697820.582267:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582275:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697820.582279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697820/real 1787697820] req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697820.582304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697820.582307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697820.582310:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697820.582322:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126261504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697820.582329:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697820.582334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126261632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697820.582341:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697820.582345:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126261760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697820.582352:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697820.582358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697820.582375:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582381:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697820.582390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697820.582396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697845 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582404:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697820.582409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697820/real 1787697820] req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697820.582458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697820.582465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697820.582468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697820.582471:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697820.582478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126261888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697820.582485:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126261888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697845 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697820.582492:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697821.093523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697821.093523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697821.093523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697821.221574:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697822.117512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697822.117512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697822.117543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697822.245590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697822.501479:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300395887, at 4300395904 00000100:00000040:3.0:1787697822.501479:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300395887, at 4300395904 00000100:00100000:0.0:1787697823.141529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697823.141529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697823.141529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697823.269589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697824.165532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697824.165532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697824.165532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697824.293577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697825.189518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697825.189518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697825.189525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697825.317562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697825.701543:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697825.701551:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697825.701554:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697825.701562:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5726 00000100:00080000:1.0:1787697825.701576:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697825.701578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701581:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701583:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697825.701587:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697825.701589:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701590:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701593:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697825.701595:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701606:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697825.701624:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697825.701639:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697825.701654:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697825.701658:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697825.701659:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697825.701663:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5726 00000100:00080000:1.0:1787697825.701666:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697825.701669:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701672:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701673:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697825.701676:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697825.701679:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701681:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701683:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697825.701686:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697825.701694:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697825.701697:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697825.701699:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697825.701706:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697825.701711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126262016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697825.701715:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697825.701722:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697825.701725:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697825.701726:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697825.701730:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5726 00000100:00080000:1.0:1787697825.701732:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697825.701735:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697825.701736:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697825.701737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701739:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697825.701742:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697825.701744:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697825.701745:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701747:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701748:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697825.701750:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697825.701754:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697825.701757:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697825.701763:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697825.701770:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697825.701776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787697825.701784:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697825.701787:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697825.701788:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697825.701790:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697825.701792:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5726 00000100:00080000:1.0:1787697825.701795:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697825.701797:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701799:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701801:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697825.701802:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697825.701805:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701807:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701809:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697825.701811:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697825.701814:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697825.701817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697825/real 1787697825] req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697825.701819:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697825.701826:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697825.701831:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697825.701834:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.701840:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697825.701848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697825.701851:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787697825.701853:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697825.701856:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:2.0:1787697825.701856:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697825.701858:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697825.701861:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5726 00000100:00080000:1.0:1787697825.701865:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697825.701865:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126262016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697825.701867:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701869:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701871:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787697825.701872:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126262016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697825.701873:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697825.701876:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697825.701878:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697825.701880:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000020:00000040:2.0:1787697825.701882:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787697825.701883:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697825.701887:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697825.701891:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697825.701895:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697825.701898:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697825.701904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697825.701912:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.701916:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697825.701921:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5736) 00000100:00000040:2.0:1787697825.701923:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697825.701934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697825.701940:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.701949:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697825.701956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697825/real 1787697825] req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.701965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.701970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697825.701975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697825.701977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697825.701979:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.701984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697825.701987:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.701991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697825.701999:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702003:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697825.702009:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697825.702013:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702019:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697825.702022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697825/real 1787697825] req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702034:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697825.702039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697825.702040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697825.702042:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702047:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697825.702050:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697825.702061:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702065:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697825.702070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697825.702073:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702078:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697825.702081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697825/real 1787697825] req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702099:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697825.702105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697825.702114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697825.702118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697825.702131:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126262144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697825.702137:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697825.702141:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126262272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697825.702145:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697825.702148:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126262400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697825.702151:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697825.702156:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697825.702169:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702173:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697825.702180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697825.702184:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697850 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702189:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697825.702192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697825/real 1787697825] req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697825.702203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697825.702208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697825.702210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697825.702212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697825.702215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697825.702218:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126262528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697850 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697825.702222:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697826.213530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697826.213530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697826.216417:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697826.341598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697827.237532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697827.237532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697827.237532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697827.365544:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697827.621523:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300401007, at 4300401024 00000100:00000040:3.0:1787697827.621523:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300401007, at 4300401024 00000100:00100000:0.0:1787697828.261492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697828.261492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697828.261493:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697828.389583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697829.285516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697829.285516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697829.285516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697829.413549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697830.309481:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697830.309487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697830.309500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697830.437572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697830.821540:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697830.821550:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697830.821555:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697830.821565:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5731 00000100:00080000:2.0:1787697830.821576:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697830.821580:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821584:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821587:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697830.821592:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697830.821596:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821598:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821602:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697830.821606:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821620:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697830.821641:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697830.821653:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697830.821671:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697830.821673:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697830.821674:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697830.821677:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5731 00000100:00080000:2.0:1787697830.821680:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697830.821682:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821684:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821685:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697830.821687:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697830.821689:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821690:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821692:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697830.821695:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697830.821704:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697830.821709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697830.821715:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697830.821722:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697830.821735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126262656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697830.821740:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697830.821752:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697830.821754:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697830.821756:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697830.821758:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5731 00000100:00080000:2.0:1787697830.821762:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697830.821763:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821765:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821766:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697830.821768:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697830.821770:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821775:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697830.821776:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697830.821777:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697830.821782:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697830.821787:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697830.821793:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697830.821794:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787697830.821803:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697830.821814:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697830.821822:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697830.821825:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697830.821827:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697830.821831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697830.821833:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5731 00000100:00080000:2.0:1787697830.821839:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697830.821843:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821847:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821852:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:1.0:1787697830.821853:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697830.821856:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697830.821864:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821869:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821872:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697830.821875:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697830.821880:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697830.821885:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697830.821890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697830/real 1787697830] req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697830.821897:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697830.821909:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697830.821911:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697830.821934:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697830.821935:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697830.821940:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697830.821945:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697830.821952:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697830.821953:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5731 00000100:00080000:2.0:1787697830.821960:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697830.821961:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697830.821967:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697830.821970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697830.821974:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697830.821979:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697830.821983:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787697830.821987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126262656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697830.821991:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697830.821999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697830.822000:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126262656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697830.822004:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697830.822011:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697830.822015:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:2.0:1787697830.822017:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697830.822023:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697830.822032:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697830.822034:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697830.822055:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697830.822060:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822073:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697830.822074:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5741) 00000100:00000040:1.0:1787697830.822084:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697830.822098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697830.822106:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822114:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697830.822120:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697830/real 1787697830] req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822140:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697830.822154:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697830.822157:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697830.822161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697830.822175:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697830.822195:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822203:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697830.822213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697830.822219:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822228:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697830.822234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697830/real 1787697830] req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697830.822298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697830.822302:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697830.822306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126262912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697830.822319:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697830.822340:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822355:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697830.822366:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697830.822373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822383:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697830.822389:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697830/real 1787697830] req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697830.822435:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697830.822440:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697830.822444:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697830.822459:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126262784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697830.822466:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697830.822472:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126262912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697830.822479:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697830.822491:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126263040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697830.822497:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697830.822503:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697830.822518:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822522:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697830.822530:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697830.822535:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697855 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822540:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697830.822543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697830/real 1787697830] req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697830.822558:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697830.822563:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697830.822564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697830.822566:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697830.822576:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697830.822579:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126263168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697855 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697830.822583:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697831.333531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697831.333531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697831.333531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697831.461557:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697832.357528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697832.357528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697832.357616:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697832.485575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697832.741514:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300406127, at 4300406144 00000100:00000040:3.0:1787697832.741514:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300406127, at 4300406144 00000100:00100000:0.0:1787697833.381505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697833.381505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697833.381513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697833.509564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697834.405521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697834.405521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697834.405543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697834.533667:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697835.429555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697835.429555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697835.429556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697835.557553:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697835.941582:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697835.941592:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697835.941596:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697835.941607:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5736 00000100:00080000:1.0:1787697835.941617:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697835.941620:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941624:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941627:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697835.941632:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697835.941634:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941638:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697835.941640:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941656:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697835.941681:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697835.941695:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697835.941726:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697835.941730:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697835.941731:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697835.941736:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5736 00000100:00080000:1.0:1787697835.941740:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697835.941743:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941746:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941749:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697835.941752:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697835.941755:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941759:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941762:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:2.0:1787697835.941767:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697835.941768:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697835.941793:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697835.941794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126263296:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787697835.941805:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697835.941816:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697835.941823:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697835.941829:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697835.941832:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697835.941833:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697835.941836:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5736 00000100:00080000:1.0:1787697835.941839:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697835.941843:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941845:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697835.941845:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697835.941849:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697835.941852:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697835.941856:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941862:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941866:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697835.941868:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697835.941873:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787697835.941875:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697835.941880:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697835.941886:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697835.941892:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697835.941897:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697835.941901:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697835.941903:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697835.941908:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5736 00000100:00080000:1.0:1787697835.941912:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787697835.941916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.941925:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941927:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697835.941930:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697835.941933:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697835.941937:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697835.941940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.941947:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697835.941952:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697835.941959:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697835.941963:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697835.941973:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697835.941984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697835/real 1787697835] req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697835.941987:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697835.942012:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697835.942019:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:2.0:1787697835.942020:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697835.942023:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697835.942033:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5736 00000100:00080000:2.0:1787697835.942033:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697835.942039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787697835.942042:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697835.942045:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697835.942049:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697835.942055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126263296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697835.942058:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.942065:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787697835.942065:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126263296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697835.942070:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000020:00000040:2.0:1787697835.942077:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697835.942078:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697835.942084:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697835.942087:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697835.942090:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697835.942095:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697835.942101:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697835.942101:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263424:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697835.942118:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697835.942130:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942136:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697835.942145:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5747) 00000100:00000040:2.0:1787697835.942149:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697835.942175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697835.942182:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942191:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697835.942195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697835/real 1787697835] req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942213:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697835.942217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697835.942219:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697835.942221:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697835.942229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697835.942242:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942247:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697835.942254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697835.942258:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942263:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697835.942266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697835/real 1787697835] req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697835.942295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697835.942299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697835.942302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697835.942316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697835.942335:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942339:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697835.942347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697835.942351:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942357:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697835.942361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697835/real 1787697835] req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697835.942381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697835.942383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697835.942385:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697835.942392:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126263424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697835.942397:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697835.942401:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126263552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697835.942405:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697835.942409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126263680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697835.942412:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697835.942417:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942445:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697835.942458:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942466:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697835.942477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697835.942484:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697860 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942501:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697835.942508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697835/real 1787697835] req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697835.942530:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697835.942538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697835.942541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697835.942545:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697835.942554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126263808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697835.942559:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126263808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697860 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697835.942568:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697836.453558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697836.453559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697836.453588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697836.581563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697837.477543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697837.477548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697837.477661:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697837.605586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697837.861485:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300411247, at 4300411264 00000100:00000040:3.0:1787697837.861488:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300411247, at 4300411264 00000100:00100000:1.0:1787697838.501512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697838.501515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697838.501521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697838.629583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697839.525541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697839.525541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697839.525541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697839.653642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697840.549524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697840.549524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697840.549530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697840.677597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697841.061512:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697841.061525:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697841.061531:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697841.061544:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5742 00000100:00080000:1.0:1787697841.061556:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697841.061560:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061565:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061569:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697841.061574:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697841.061578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061580:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061584:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697841.061588:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061605:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697841.061635:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697841.061651:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.061676:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697841.061680:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697841.061681:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697841.061686:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5742 00000100:00080000:1.0:1787697841.061689:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697841.061692:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061695:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061697:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697841.061700:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697841.061704:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061707:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061709:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697841.061713:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697841.061723:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697841.061728:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697841.061729:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697841.061742:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697841.061750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126263936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697841.061758:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.061767:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697841.061771:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697841.061773:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697841.061777:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5742 00000100:00000040:2.0:1787697841.061787:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.061790:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697841.061801:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697841.061803:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061811:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697841.061813:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697841.061817:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061821:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061823:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697841.061826:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697841.061831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697841.061836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697841.061846:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787697841.061851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061862:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.061871:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697841.061877:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:2.0:1787697841.061878:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.061879:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697841.061889:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5742 00000100:00080000:1.0:1787697841.061895:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697841.061898:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061901:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061903:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697841.061905:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697841.061909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.061912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.061914:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697841.061917:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697841.061923:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697841.061925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697841.061928:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697841.061942:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697841.061949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697841.061959:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.061972:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697841.061979:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697841.061982:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697841.061988:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5742 00000100:00000040:2.0:1787697841.061991:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.061993:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697841.061999:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697841.062005:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.062010:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:2.0:1787697841.062011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787697841.062013:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697841.062020:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697841.062022:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697841.062026:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697841.062030:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787697841.062031:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697841.062035:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697841.062044:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697841.062049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126263936:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787697841.062051:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697841.062064:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697841.062067:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126263936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697841.062098:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697841.062101:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697841.062115:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5752) 00000100:00000040:2.0:1787697841.062125:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062137:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697841.062151:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062158:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697841.062170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697841.062177:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062188:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697841.062193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697841.062219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697841.062222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697841.062226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697841.062244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697841.062264:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062270:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697841.062280:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697841.062287:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062295:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697841.062300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697841.062335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697841.062338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697841.062342:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062349:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697841.062355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697841.062374:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062381:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697841.062391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697841.062397:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062405:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697841.062410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697841.062461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697841.062464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697841.062468:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697841.062481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062489:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697841.062501:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062508:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697841.062518:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697841.062532:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697866 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062542:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697841.062547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697841/real 1787697841] req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697841.062565:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697841.062572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697841.062575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697841.062579:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697841.062586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697841.062591:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126264064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697841.062598:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697841.062605:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126264192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697841.062611:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697841.062616:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126264320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697841.062623:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697841.062627:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126264448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697866 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697841.062634:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697841.573549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697841.573549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697841.573731:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697841.701629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697842.597572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697842.597572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697842.597572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697842.725621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697842.981520:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300416367, at 4300416384 00000100:00000040:3.0:1787697842.981520:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300416367, at 4300416384 00000100:00100000:1.0:1787697843.621569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697843.621569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697843.621570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697843.749598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697844.645513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697844.645513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697844.645518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697844.773577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697845.669504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697845.669504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697845.669504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697845.797583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697846.181563:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697846.181577:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697846.181582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697846.181596:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5747 00000100:00080000:3.0:1787697846.181610:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697846.181614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697846.181619:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181623:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697846.181629:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697846.181632:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697846.181635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181638:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697846.181642:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181657:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697846.181682:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697846.181694:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697846.181713:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697846.181716:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697846.181718:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697846.181721:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5747 00000100:00080000:3.0:1787697846.181723:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697846.181725:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697846.181743:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181746:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697846.181748:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697846.181750:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697846.181753:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181756:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697846.181758:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697846.181766:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697846.181769:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697846.181779:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697846.181787:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.181795:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697846.181796:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697846.181805:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697846.181808:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697846.181820:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5747 00000100:00080000:3.0:1787697846.181830:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697846.181839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697846.181840:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126264576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697846.181851:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181855:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697846.181857:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697846.181861:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697846.181863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181865:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697846.181868:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697846.181884:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697846.181886:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697846.181891:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697846.181905:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697846.181911:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697846.181923:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697846.181934:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697846.181938:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697846.181941:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787697846.181948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697846.181953:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5747 00000100:00080000:3.0:1787697846.181961:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697846.181966:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697846.181971:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697846.181973:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.181981:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697846.181997:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697846.182006:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697846.182012:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.182015:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697846.182018:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697846.182025:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697846.182030:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697846.182041:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697846/real 1787697846] req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697846.182045:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697846.182098:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697846.182174:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697846.182183:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697846.182187:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697846.182193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787697846.182195:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5747 00000100:00080000:3.0:1787697846.182204:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697846.182207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787697846.182213:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697846.182222:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697846.182225:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.182234:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:2.0:1787697846.182248:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126264576:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787697846.182248:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697846.182261:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697846.182265:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126264576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697846.182270:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697846.182276:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697846.182288:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697846.182297:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:3.0:1787697846.182297:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697846.182307:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697846.182324:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697846.182325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697846.182367:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697846.182383:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5757) 00000100:00100000:2.0:1787697846.182385:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697846.182406:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182414:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697846.182454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697846.182463:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182473:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697846.182479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697846/real 1787697846] req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182507:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697846.182513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697846.182517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697846.182520:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182528:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697846.182535:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697846.182555:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182563:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697846.182573:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697846.182579:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182587:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697846.182592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697846/real 1787697846] req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182608:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182615:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697846.182623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697846.182626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697846.182630:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697846.182643:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697846.182663:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182670:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697846.182680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697846.182687:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182696:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697846.182701:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697846/real 1787697846] req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182726:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697846.182733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697846.182736:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697846.182740:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126264960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697846.182752:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126264704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697846.182760:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697846.182766:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126264832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697846.182774:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697846.182779:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126264960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697846.182785:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697846.182793:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182801:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697846.182814:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182820:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697846.182831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697846.182837:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697871 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182846:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697846.182851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697846/real 1787697846] req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697846.182869:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697846.182875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697846.182878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697846.182882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697846.182889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697846.182895:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126265088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697871 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697846.182901:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697846.693540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697846.693541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697846.693541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697846.821583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697847.717539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697847.717539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697847.717553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697847.845631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697848.101549:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300421487, at 4300421504 00000100:00000040:3.0:1787697848.101558:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300421487, at 4300421504 00000100:00100000:0.0:1787697848.741567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697848.741567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697848.741587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697848.869629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697849.765517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697849.765517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697849.765517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697849.893606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697850.789518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697850.789524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697850.789558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697850.917575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697851.301555:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697851.301567:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697851.301572:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697851.301584:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5752 00000100:00080000:1.0:1787697851.301596:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697851.301600:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.301605:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301609:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697851.301614:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697851.301618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.301621:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301624:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697851.301628:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301644:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697851.301679:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697851.301695:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697851.301716:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697851.301720:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697851.301722:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697851.301726:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5752 00000100:00080000:1.0:1787697851.301730:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697851.301734:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.301737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301740:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697851.301743:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697851.301746:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.301749:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301752:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697851.301758:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697851.301759:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787697851.301768:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697851.301774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126265216:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787697851.301775:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697851.301796:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697851.301806:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.301815:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697851.301816:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697851.301824:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697851.301828:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697851.301835:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5752 00000100:00000040:2.0:1787697851.301835:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787697851.301840:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697851.301844:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.301847:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301849:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697851.301852:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697851.301855:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.301858:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301860:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697851.301864:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697851.301869:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697851.301873:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787697851.301878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787697851.301886:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697851.301912:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697851.301913:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697851.301937:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697851.301941:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697851.301943:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697851.301948:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5752 00000100:00080000:1.0:1787697851.301952:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697851.301956:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697851.301960:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697851/real 1787697851] req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697851.301963:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.301969:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697851.301973:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697851.301978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.301984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697851.301984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697851.301990:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697851.301996:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697851.302001:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697851.302010:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697851.302011:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787697851.302021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697851.302025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:1.0:1787697851.302026:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697851.302030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126265216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697851.302060:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302069:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126265216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697851.302077:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697851.302082:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697851.302084:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:2.0:1787697851.302086:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787697851.302090:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5752 00000100:00080000:1.0:1787697851.302096:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697851.302099:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.302101:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697851.302104:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697851.302108:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697851.302108:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697851.302116:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697851.302124:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697851.302124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265344:192.168.201.136@tcp:38:kworker.0 00000020:00000040:1.0:1787697851.302128:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697851.302133:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697851.302137:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697851.302142:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697851.302145:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302155:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787697851.302156:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697851.302170:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697851.302170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697851.302182:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697851.302185:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5762) 00000100:00000040:2.0:1787697851.302203:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697851.302209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697851/real 1787697851] req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302222:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697851.302227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697851.302229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697851.302231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697851.302244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697851.302257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302261:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697851.302268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697851.302272:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302277:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697851.302281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697851/real 1787697851] req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697851.302297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697851.302298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697851.302300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697851.302308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697851.302321:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302324:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697851.302329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697851.302332:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302339:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697851.302344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697851/real 1787697851] req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697851.302370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697851.302373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697851.302377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697851.302391:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126265344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697851.302400:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697851.302406:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126265472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697851.302414:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697851.302439:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126265600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697851.302451:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697851.302458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302463:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697851.302472:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302477:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697851.302485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697851.302490:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697876 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302495:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697851.302498:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697851/real 1787697851] req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697851.302518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697851.302531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697851.302535:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697851.302539:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697851.302546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697851.302552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126265728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697876 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697851.302560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697851.813510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697851.813510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697851.813510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697851.941611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697852.837543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697852.837544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697852.837544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697852.965553:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697853.221516:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300426607, at 4300426624 00000100:00000040:3.0:1787697853.221516:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300426607, at 4300426624 00000100:00100000:1.0:1787697853.861534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697853.861534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697853.861540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697853.989625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697854.885524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697854.885524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697854.885524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697855.013609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697855.909491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697855.909491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697855.909503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697856.037610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697856.421540:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697856.421573:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697856.421578:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697856.421591:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5757 00000100:00080000:3.0:1787697856.421603:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697856.421607:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.421611:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.421616:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697856.421621:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697856.421625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.421628:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.421631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697856.421635:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697856.421649:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697856.421679:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697856.421707:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697856.421731:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697856.421735:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697856.421737:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697856.421741:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5757 00000100:00080000:3.0:1787697856.421745:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697856.421748:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.421752:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.421755:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697856.421759:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697856.421768:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.421771:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.421774:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:2.0:1787697856.421778:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697856.421779:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697856.421794:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697856.421795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126265856:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787697856.421802:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697856.421834:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697856.421846:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697856.421854:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697856.421859:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697856.421865:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697856.421866:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697856.421870:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697856.421875:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5757 00000100:00080000:3.0:1787697856.421891:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697856.421895:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.421898:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.421901:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697856.421904:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697856.421906:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.421910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.421912:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697856.421915:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697856.421920:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697856.421926:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697856.421939:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697856.421945:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697856.421974:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787697856.421979:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697856.421992:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697856.421992:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697856.422001:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697856.422006:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697856.422007:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697856.422015:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5757 00000100:00000040:2.0:1787697856.422020:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787697856.422020:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697856.422026:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697856.422030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697856.422033:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.422040:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697856.422045:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787697856.422047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787697856.422051:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.422056:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.422059:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697856.422062:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697856.422066:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697856.422069:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697856.422078:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697856.422083:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697856.422093:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697856.422099:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787697856.422108:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697856.422121:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697856.422124:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697856.422126:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697856.422136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697856.422138:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5757 00000100:00080000:3.0:1787697856.422147:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697856.422150:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787697856.422155:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697856.422160:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.422163:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697856.422166:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697856.422167:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697856.422174:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697856.422179:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697856.422182:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697856.422186:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697856.422189:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697856.422192:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000008:00100000:3.0:1787697856.422194:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697856.422201:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697856.422212:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697856.422222:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697856.422248:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5767) 00000100:00000040:2.0:1787697856.422307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697856.422317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697856.422332:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697856.422341:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697856.422356:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697856.933491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697856.933491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697856.933492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697857.061615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697857.957532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697857.957539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697857.957563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697858.085610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697858.341576:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300431727, at 4300431744 00000100:00000040:3.0:1787697858.341576:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300431727, at 4300431744 00000100:00100000:2.0:1787697858.981559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697858.981582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697858.981588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697859.109573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787697859.493701:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787697859.493743:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697859.493772:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697859.493808:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697859.493837:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697859.493846:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697859.493860:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697859.493868:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697859.493877:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697859.493884:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697859.493892:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697859.493899:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697881 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.493974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697856/real 1787697859] req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494011:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697859.494022:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697859.494034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697859.494039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697859.494044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494051:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266368:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787697859.494060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697856/real 1787697859] req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697859.494078:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697859.494096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697859.494101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697859.494108:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494115:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266240:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787697859.494123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697856/real 1787697859] req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697859.494142:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697859.494151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697859.494154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697859.494158:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266112:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787697859.494173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697856/real 1787697859] req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697859.494193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697859.494220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697859.494224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697859.494228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126265984:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787697859.494245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697856/real 1787697859] req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697859.494270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697859.494283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697859.494287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697859.494291:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697859.494298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126265856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697859.494306:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126266368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697859.494330:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697859.494343:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126266240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697859.494358:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697859.494365:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126266112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697859.494372:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697859.494378:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126265984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697859.494385:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697859.494390:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126265856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697881 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697859.494397:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697860.005555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697860.005556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697860.005562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697860.133587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697861.029520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697861.029531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697861.029552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697861.157597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697861.541539:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697861.541549:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697861.541553:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697861.541562:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5762 00000100:00080000:1.0:1787697861.541571:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697861.541575:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541579:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541582:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697861.541588:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697861.541591:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541593:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541596:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697861.541598:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541608:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697861.541628:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697861.541642:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697861.541677:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697861.541681:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697861.541682:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697861.541687:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5762 00000100:00080000:1.0:1787697861.541690:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697861.541693:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541696:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541698:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697861.541700:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697861.541703:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541705:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541707:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697861.541709:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697861.541718:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697861.541734:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697861.541746:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697861.541754:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697861.541762:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697861.541765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697861.541766:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697861.541770:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5762 00000100:00080000:1.0:1787697861.541773:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697861.541776:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541780:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697861.541782:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697861.541784:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541787:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541789:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697861.541791:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697861.541796:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697861.541800:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697861.541807:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697861.541816:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697861.541835:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697861.541838:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697861.541840:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697861.541844:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5762 00000100:00080000:1.0:1787697861.541847:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697861.541849:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541852:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541854:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697861.541856:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697861.541858:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541861:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541863:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697861.541866:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697861.541870:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697861.541875:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697861.541883:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697861.541892:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697861.541899:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697861.541902:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697861.541904:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697861.541908:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5762 00000100:00080000:1.0:1787697861.541911:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697861.541913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541916:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541919:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697861.541920:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697861.541923:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697861.541926:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697861.541927:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697861.541930:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697861.541934:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697861.541938:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697861.541945:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697861.541954:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697861.541964:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5772) 00000100:00000040:2.0:1787697861.542039:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126266496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697861.542081:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542089:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697861.542121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697861.542134:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697861/real 1787697861] req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542187:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697861.542196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697861.542199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697861.542203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126266496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697861.542217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542225:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697861.542237:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542244:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697861.542253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697861.542260:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542269:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697861.542274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697861/real 1787697861] req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697861.542300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697861.542303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697861.542307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697861.542325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697861.542345:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542352:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697861.542362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697861.542368:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542377:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697861.542382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697861/real 1787697861] req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542401:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697861.542408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697861.542412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697861.542415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697861.542450:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542458:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697861.542471:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542477:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697861.542487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697861.542493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542501:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697861.542506:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697861/real 1787697861] req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542516:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542524:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697861.542531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697861.542534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697861.542538:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126266880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697861.542552:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542559:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697861.542570:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542576:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697861.542585:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697861.542591:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697886 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542599:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697861.542604:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697861/real 1787697861] req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697861.542622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697861.542628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697861.542631:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697861.542635:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697861.542641:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697861.542647:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126266496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697861.542669:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697861.542677:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126266624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697861.542684:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697861.542689:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126266752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697861.542695:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697861.542700:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126266880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697861.542706:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697861.542710:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126267008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697886 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697861.542716:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697862.053527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697862.053527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697862.053532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697862.181546:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697863.077511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697863.077511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697863.077511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697863.205585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697863.461536:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300436847, at 4300436864 00000100:00000040:3.0:1787697863.461598:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300436847, at 4300436864 00000100:00100000:0.0:1787697864.101571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697864.101571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697864.101572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697864.229618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697865.125534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697865.125534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697865.125563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697865.253587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697866.149508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697866.149508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697866.149508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697866.277603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697866.661530:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697866.661537:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697866.661540:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697866.661548:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5767 00000100:00080000:3.0:1787697866.661556:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697866.661558:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661561:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661563:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697866.661567:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697866.661569:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661571:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661573:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697866.661575:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661586:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697866.661604:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697866.661614:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697866.661633:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697866.661652:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697866.661654:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697866.661658:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5767 00000100:00080000:3.0:1787697866.661661:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697866.661664:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661668:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697866.661683:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697866.661691:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697866.661695:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00100000:2.0:1787697866.661699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126267136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697866.661700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661706:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661710:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697866.661712:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697866.661729:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697866.661731:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697866.661737:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697866.661747:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697866.661754:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697866.661777:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697866.661781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697866.661793:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697866.661798:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787697866.661799:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697866.661801:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697866.661810:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5767 00000100:00080000:3.0:1787697866.661817:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697866.661819:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661823:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661825:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697866.661827:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00100000:2.0:1787697866.661832:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697866/real 1787697866] req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697866.661833:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661840:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661846:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697866.661850:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697866.661852:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697866.661859:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697866.661861:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697866.661868:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697866.661871:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:3.0:1787697866.661872:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697866.661875:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.661882:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126267136:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787697866.661886:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697866.661889:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126267136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697866.661914:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661919:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.661930:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697866.661931:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697866.661938:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697866.661941:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697866.661950:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5767 00000100:00100000:2.0:1787697866.661951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267264:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697866.661954:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697866.661959:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661962:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661964:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697866.661967:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697866.661969:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697866.661972:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.661978:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.661983:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787697866.661989:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697866.661990:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697866.661997:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697866.662003:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000800:00000100:2.0:1787697866.662008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697866.662016:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697866.662020:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697866.662037:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662040:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787697866.662050:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697866.662051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697866/real 1787697866] req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697866.662062:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697866.662066:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697866.662071:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5767 00000100:00080000:3.0:1787697866.662076:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697866.662079:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.662083:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697866.662084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697866.662085:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697866.662089:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697866.662095:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697866.662101:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697866.662104:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787697866.662106:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697866.662108:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697866.662116:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787697866.662121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787697866.662122:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787697866.662130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00100000:3.0:1787697866.662135:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697866.662145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697866.662148:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787697866.662172:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5777) 00000100:00000040:2.0:1787697866.662179:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126267264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697866.662190:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697866.662200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697866.662222:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662230:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697866.662243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697866.662251:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662259:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697866.662266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697866/real 1787697866] req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697866.662294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697866.662297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697866.662301:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697866.662315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697866.662336:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662344:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697866.662356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697866.662363:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662378:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697866.662384:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697866/real 1787697866] req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697866.662411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697866.662413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697866.662417:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697866.662453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697866.662473:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662477:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697866.662485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697866.662490:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697891 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662496:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697866.662499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697866/real 1787697866] req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697866.662517:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697866.662524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697866.662526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697866.662531:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697866.662538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697866.662544:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126267392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697866.662552:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697866.662558:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126267520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697866.662564:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697866.662569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126267648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697891 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697866.662576:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697867.173566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697867.173566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697867.173569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697867.301667:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697868.197523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697868.197524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697868.197658:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697868.325570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697868.581516:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300441967, at 4300441984 00000100:00000040:3.0:1787697868.581516:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300441967, at 4300441984 00000100:00100000:1.0:1787697869.221556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697869.221556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697869.221687:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697869.349568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697870.245593:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697870.245595:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697870.245629:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697870.373614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697871.269525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697871.269525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697871.269538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697871.397607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697871.781560:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697871.781567:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697871.781587:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697871.781596:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5772 00000100:00080000:3.0:1787697871.781605:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697871.781607:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781613:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697871.781617:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697871.781620:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781622:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781624:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697871.781627:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781636:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697871.781660:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697871.781675:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697871.781698:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697871.781702:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697871.781703:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697871.781708:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5772 00000100:00080000:3.0:1787697871.781712:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697871.781714:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781717:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781720:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697871.781723:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697871.781726:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781730:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697871.781738:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697871.781739:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697871.781745:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697871.781756:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697871.781764:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697871.781771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126267776:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787697871.781779:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697871.781791:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.781800:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697871.781802:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697871.781811:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697871.781815:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697871.781821:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697871.781823:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5772 00000100:00080000:3.0:1787697871.781829:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697871.781833:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781836:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781838:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697871.781840:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697871.781841:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781843:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781845:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697871.781847:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697871.781851:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697871.781855:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787697871.781860:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697871.781867:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697871.781880:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697871.781887:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697871.781899:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697871.781906:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697871.781907:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697871.781911:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5772 00000100:00080000:3.0:1787697871.781914:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697871.781916:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781917:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781919:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697871.781921:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697871.781923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.781924:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.781926:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697871.781928:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697871.781932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697871/real 1787697871] req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697871.781932:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697871.781938:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697871.781955:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697871.781959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697871.781978:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.781984:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697871.781987:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697871.781993:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697871.781997:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697871.782002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697871.782007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697871.782014:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697871.782016:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5772 00000100:00080000:3.0:1787697871.782024:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697871.782030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126267776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697871.782031:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.782038:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.782042:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787697871.782043:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126267776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697871.782045:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697871.782051:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697871.782056:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697871.782060:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697871.782065:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697871.782068:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:3.0:1787697871.782070:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697871.782075:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697871.782087:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697871.782088:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697871.782120:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787697871.782142:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5782) 00000100:00000040:2.0:1787697871.782148:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782158:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697871.782169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697871.782176:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782198:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697871.782204:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697871/real 1787697871] req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782223:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697871.782230:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697871.782233:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697871.782238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126267904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697871.782251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697871.782272:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782280:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697871.782289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697871.782296:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782305:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697871.782310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697871/real 1787697871] req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697871.782338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697871.782341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697871.782346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697871.782359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697871.782379:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782386:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697871.782406:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697871.782413:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782444:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697871.782450:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697871/real 1787697871] req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782471:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697871.782479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697871.782482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697871.782486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697871.782500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126267904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697871.782508:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697871.782513:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126268032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697871.782521:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697871.782526:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126268160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697871.782534:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697871.782541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782562:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697871.782574:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782582:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697871.782592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697871.782599:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697896 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782608:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697871.782613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697871/real 1787697871] req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782623:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697871.782632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697871.782639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697871.782642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697871.782646:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697871.782654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697871.782662:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126268288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697896 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697871.782670:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697872.293512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697872.293551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697872.293809:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697872.421690:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697873.317551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697873.317551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697873.317551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697873.445663:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697873.701527:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300447087, at 4300447104 00000100:00000040:3.0:1787697873.701530:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300447087, at 4300447104 00000100:00100000:3.0:1787697874.341571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697874.341574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697874.341577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697874.469577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697875.365567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697875.365574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697875.365800:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697875.493578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697876.389542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697876.389542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697876.389543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697876.517559:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697876.901572:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697876.901581:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697876.901585:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697876.901608:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5777 00000100:00080000:0.0:1787697876.901618:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697876.901621:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697876.901625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.901628:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697876.901632:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697876.901635:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697876.901638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.901641:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697876.901644:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697876.901653:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697876.901674:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697876.901688:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697876.901711:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697876.901715:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697876.901716:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697876.901720:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5777 00000100:00080000:0.0:1787697876.901723:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697876.901726:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697876.901729:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.901731:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697876.901733:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697876.901736:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697876.901739:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.901747:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697876.901750:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697876.901759:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697876.901759:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697876.901767:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697876.901776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126268416:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787697876.901782:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697876.901794:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.901803:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697876.901806:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697876.901815:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787697876.901819:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697876.901820:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697876.901826:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5777 00000100:00080000:0.0:1787697876.901830:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697876.901833:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697876.901835:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.901838:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697876.901840:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697876.901844:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:2.0:1787697876.901848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697876.901852:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.901855:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697876.901859:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697876.901863:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787697876.901864:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697876.901871:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697876.901882:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697876.901892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697876/real 1787697876] req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697876.901927:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.901928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697876.901952:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787697876.901953:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697876.901957:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697876.901961:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697876.901971:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5777 00000100:00080000:2.0:1787697876.901974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697876.901976:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697876.901984:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697876.901984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697876.901991:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697876.901994:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697876.901995:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697876.901999:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697876.902007:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697876.902013:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697876.902013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126268416:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697876.902018:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697876.902023:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697876.902027:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126268416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697876.902030:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697876.902037:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:2.0:1787697876.902042:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697876.902049:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697876.902060:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902063:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697876.902084:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787697876.902086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268544:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787697876.902092:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697876.902096:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697876.902106:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5777 00000100:00000040:2.0:1787697876.902111:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697876.902118:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697876.902125:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697876.902125:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697876.902131:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.902134:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697876.902137:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697876.902140:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:2.0:1787697876.902146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697876.902148:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697876.902152:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697876.902158:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697876.902159:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697876.902165:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697876.902171:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697876.902172:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697876.902181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697876/real 1787697876] req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697876.902186:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697876.902198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697876.902204:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697876.902221:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5788) 00000100:00080000:2.0:1787697876.902234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697876.902238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697876.902244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697876.902256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697876.902282:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902289:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697876.902299:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697876.902306:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902314:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697876.902320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697876/real 1787697876] req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697876.902344:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697876.902347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697876.902351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697876.902361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126268544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697876.902368:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697876.902373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126268672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697876.902380:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697876.902387:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697876.902405:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902412:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697876.902448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697876.902454:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902480:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697876.902486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697876/real 1787697876] req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902516:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697876.902524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697876.902527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697876.902531:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902538:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697876.902543:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697876.902561:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902568:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697876.902578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697876.902584:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697901 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902592:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697876.902597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697876/real 1787697876] req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697876.902618:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697876.902625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697876.902628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697876.902632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697876.902639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126268928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697876.902644:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126268800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697876.902651:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697876.902657:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126268928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697901 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697876.902663:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697877.413507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697877.413525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697877.413545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697877.541607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697878.437490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697878.437518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697878.437518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697878.565667:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697878.821551:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300452207, at 4300452224 00000100:00000040:3.0:1787697878.821561:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300452207, at 4300452224 00000100:00100000:1.0:1787697879.461564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697879.461564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697879.461565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697879.589594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697880.485567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697880.485567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697880.485568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697880.613655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697881.509507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697881.509508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697881.509525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697881.637622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697882.021558:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697882.021572:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697882.021577:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697882.021591:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5783 00000100:00080000:0.0:1787697882.021604:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697882.021609:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021614:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021618:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697882.021624:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697882.021649:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021651:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021654:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697882.021656:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021668:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697882.021694:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697882.021706:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697882.021732:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697882.021735:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697882.021736:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697882.021739:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5783 00000100:00080000:0.0:1787697882.021743:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697882.021745:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021746:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021748:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697882.021750:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697882.021752:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021753:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021755:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697882.021756:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697882.021765:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697882.021769:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697882.021780:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697882.021781:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.021809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126269056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697882.021824:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697882.021835:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697882.021838:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697882.021840:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697882.021845:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5783 00000100:00000040:2.0:1787697882.021850:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697882.021851:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697882.021859:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021866:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021870:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:2.0:1787697882.021871:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021876:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697882.021880:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021882:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021884:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697882.021886:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697882.021890:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697882.021894:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697882.021903:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697882.021909:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697882.021917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697882.021918:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697882.021923:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697882.021930:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697882.021935:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5783 00000100:00080000:0.0:1787697882.021940:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697882.021945:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697882.021946:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021953:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021958:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697882.021963:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697882.021968:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.021971:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.021974:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697882.021976:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697882.021982:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697882.021987:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697882.021991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697882] req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697882.022001:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697882.022009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697882.022019:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697882.022038:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697882.022041:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697882.022043:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697882.022051:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5783 00000100:00080000:2.0:1787697882.022053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697882.022056:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697882.022058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697882.022065:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697882.022066:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697882.022074:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697882.022078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126269056:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697882.022080:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697882.022085:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697882.022087:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126269056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697882.022096:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697882.022097:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697882.022103:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697882.022105:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697882.022108:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697882.022112:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697882.022115:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697882.022124:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697882.022125:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269184:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697882.022139:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697882.022145:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697882.022161:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022164:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697882.022178:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5793) 00000800:00000100:2.0:1787697882.022189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697882.022198:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022207:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697882.022211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697882] req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697882.022232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697882.022236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697882.022240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697882.022253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697882.022274:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022282:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697882.022293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697882.022299:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022306:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697882.022311:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697882] req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697882.022336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697882.022339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697882.022343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697882.022356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697882.022405:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022414:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697882.022446:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697882.022453:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022463:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697882.022469:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697882] req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022496:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697882.022504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697882.022508:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697882.022512:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022519:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697882.022525:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126269184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697882.022534:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697882.022540:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126269312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697882.022547:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697882.022552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126269440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697882.022559:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697882.022566:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022571:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697882.022582:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022587:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697882.022595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697882.022599:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697907 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022604:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697882.022607:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697882/real 1787697882] req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697882.022629:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697882.022633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697882.022635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697882.022638:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697882.022642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697882.022645:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126269568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697907 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697882.022649:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697882.533545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697882.533545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697882.533545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697882.661616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697883.557539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697883.557542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697883.557566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697883.685706:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697883.941532:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300457327, at 4300457344 00000100:00000040:3.0:1787697883.941532:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300457327, at 4300457344 00000100:00100000:3.0:1787697884.581562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697884.581563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697884.582414:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697884.709815:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697885.605528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697885.605528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697885.605528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697885.733604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697886.629526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697886.629526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697886.629530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697886.757624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697887.141526:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697887.141536:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697887.141540:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697887.141557:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5788 00000100:00080000:2.0:1787697887.141568:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697887.141571:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141574:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141577:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697887.141583:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697887.141585:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141587:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141589:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697887.141592:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141607:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697887.141635:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697887.141647:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697887.141667:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697887.141670:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697887.141672:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697887.141676:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5788 00000100:00080000:2.0:1787697887.141680:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697887.141683:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141686:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141689:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697887.141692:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697887.141695:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141698:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141700:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697887.141710:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697887.141721:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697887.141722:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697887.141730:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697887.141743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126269696:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787697887.141747:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697887.141759:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697887.141769:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697887.141772:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697887.141773:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697887.141778:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5788 00000100:00000040:1.0:1787697887.141780:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697887.141783:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697887.141790:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141795:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697887.141799:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141800:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697887.141804:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697887.141808:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141811:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141813:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697887.141817:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697887.141822:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697887.141827:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697887.141836:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697887.141845:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697887.141850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697887.141859:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697887.141863:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697887.141865:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697887.141871:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5788 00000100:00000040:1.0:1787697887.141876:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697887.141877:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697887.141884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141889:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141893:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697887.141896:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697887.141898:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141900:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141901:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697887.141903:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697887.141907:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697887.141910:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697887.141919:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697887.141925:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.141932:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697887.141933:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697887.141938:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697887.141940:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697887.141955:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5788 00000100:00100000:1.0:1787697887.141959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697887.141961:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697887.141968:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141974:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141978:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787697887.141980:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697887.141982:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697887.141990:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697887.141997:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697887.141999:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697887.142000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697887.142003:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697887.142009:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697887.142010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:2.0:1787697887.142015:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697887.142024:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697887.142027:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697887.142036:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126269696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697887.142056:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5798) 00000100:00000040:1.0:1787697887.142068:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126269696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697887.142080:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697887.142103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697887.142126:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142141:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697887.142153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697887.142159:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142170:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697887.142175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142199:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697887.142206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697887.142208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697887.142212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697887.142224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697887.142241:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142248:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697887.142256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697887.142262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142270:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697887.142274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697887.142319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697887.142322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697887.142326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126269952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697887.142339:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697887.142359:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142365:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697887.142375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697887.142381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142390:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697887.142394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142449:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697887.142456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697887.142460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697887.142464:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697887.142476:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697887.142495:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142503:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697887.142512:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697887.142518:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697912 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142527:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697887.142532:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697887/real 1787697887] req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697887.142553:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697887.142560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697887.142563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697887.142567:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697887.142573:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697887.142579:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126269824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697887.142587:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697887.142593:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126269952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697887.142600:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697887.142605:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126270080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697887.142611:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697887.142616:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126270208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697912 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697887.142623:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697887.653549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697887.653550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697887.653550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697887.781612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697888.677526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697888.677526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697888.677526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697888.805606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697889.061531:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300462447, at 4300462464 00000100:00000040:3.0:1787697889.061531:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300462447, at 4300462464 00000100:00100000:2.0:1787697889.701554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697889.701555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697889.701559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697889.829608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697890.725566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697890.725566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697890.725568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697890.853568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697891.749544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697891.749545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697891.749545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697891.877550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697892.261595:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697892.261603:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697892.261606:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697892.261618:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5793 00000100:00080000:3.0:1787697892.261627:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697892.261629:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261632:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261634:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697892.261638:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697892.261641:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261644:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261647:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697892.261650:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261664:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697892.261692:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697892.261722:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697892.261747:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697892.261751:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697892.261759:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697892.261764:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5793 00000100:00080000:3.0:1787697892.261768:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697892.261770:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261773:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261774:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697892.261776:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697892.261778:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261781:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261782:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697892.261784:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697892.261794:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697892.261798:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697892.261803:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697892.261809:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697892.261833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126270336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697892.261833:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697892.261847:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697892.261850:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697892.261852:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697892.261865:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697892.261865:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5793 00000100:00080000:3.0:1787697892.261870:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697892.261875:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261879:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697892.261884:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261884:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697892.261889:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697892.261895:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261897:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261899:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697892.261902:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697892.261907:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697892.261913:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697892.261923:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697892.261932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261937:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697892.261947:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697892.261951:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:1.0:1787697892.261953:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697892.261954:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697892.261963:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5793 00000100:00080000:3.0:1787697892.261972:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697892.261975:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.261978:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.261980:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697892.261983:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697892.261987:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697892.261992:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697892/real 1787697892] req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697892.261999:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.262005:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697892.262010:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697892.262019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262034:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697892.262035:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697892.262043:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697892.262048:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697892.262053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697892.262057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697892.262064:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697892.262070:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126270336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697892.262081:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262090:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126270336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697892.262093:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697892.262100:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697892.262104:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:1.0:1787697892.262108:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787697892.262111:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5793 00000100:00080000:3.0:1787697892.262116:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697892.262119:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.262122:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.262124:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697892.262127:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697892.262128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697892.262133:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697892.262140:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697892.262143:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787697892.262146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270464:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697892.262149:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697892.262156:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697892.262162:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697892.262167:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262175:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697892.262181:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697892.262192:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697892.262215:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697892.262223:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262241:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787697892.262244:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5803) 00000100:00100000:1.0:1787697892.262255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697892/real 1787697892] req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262278:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697892.262286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697892.262289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697892.262293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697892.262308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697892.262322:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262326:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697892.262334:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697892.262339:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262344:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697892.262347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697892/real 1787697892] req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697892.262364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697892.262365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697892.262367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697892.262375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697892.262386:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262390:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697892.262394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697892.262398:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262403:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697892.262406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697892/real 1787697892] req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262417:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697892.262455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697892.262457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697892.262459:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697892.262467:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126270464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697892.262475:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697892.262481:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126270592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697892.262487:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697892.262492:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126270720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697892.262498:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697892.262505:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697892.262523:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262529:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697892.262539:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697892.262545:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697917 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262553:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697892.262559:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697892/real 1787697892] req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697892.262575:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697892.262583:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697892.262585:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697892.262587:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697892.262592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126270848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697892.262595:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126270848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697917 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697892.262599:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697892.773587:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697892.773587:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697892.773588:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697892.901587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697893.797497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697893.797499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697893.797536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697893.925575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697894.181501:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300467567, at 4300467584 00000100:00000040:2.0:1787697894.181507:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300467567, at 4300467584 00000100:00100000:1.0:1787697894.821541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697894.821564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697894.821566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697894.949569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697895.845487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697895.845487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697895.845501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697895.973562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697896.869521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697896.869521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697896.869632:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697896.997620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697897.381522:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697897.381532:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697897.381536:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697897.381547:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5798 00000100:00080000:2.0:1787697897.381557:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697897.381561:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697897.381566:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381569:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697897.381574:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697897.381578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697897.381581:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381584:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697897.381587:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381602:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697897.381628:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697897.381643:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697897.381669:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697897.381673:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697897.381675:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697897.381679:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5798 00000100:00080000:2.0:1787697897.381683:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697897.381687:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697897.381690:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381693:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697897.381695:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697897.381704:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697897.381708:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381711:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697897.381714:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697897.381725:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697897.381727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697897.381733:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697897.381745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126270976:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787697897.381752:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697897.381765:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.381775:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697897.381777:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697897.381787:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697897.381790:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697897.381791:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697897.381799:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5798 00000100:00080000:2.0:1787697897.381804:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697897.381807:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697897.381810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381812:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697897.381815:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697897.381819:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697897.381822:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381825:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000800:00000100:1.0:1787697897.381829:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697897.381833:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697897.381839:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697897.381845:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697897.381846:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697897.381860:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697897.381873:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.381878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697897/real 1787697897] req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.381896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697897.381897:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697897.381908:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697897.381912:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697897.381916:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697897.381921:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5798 00000100:00080000:1.0:1787697897.381926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697897.381929:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697897.381931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697897.381937:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697897.381938:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697897.381947:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697897.381952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126270976:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787697897.381953:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697897.381958:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697897.381959:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126270976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697897.381967:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697897.381972:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697897.381974:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.381979:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697897.381987:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697897.381990:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787697897.381996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697897.382005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271104:192.168.201.136@tcp:38:kworker.0 00000020:00000040:2.0:1787697897.382005:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697897.382019:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697897.382028:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697897.382036:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382048:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697897.382056:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697897.382060:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697897.382062:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787697897.382064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697897.382070:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5798 00000100:00000040:1.0:1787697897.382075:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697897.382077:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697897.382085:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697897.382090:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:2.0:1787697897.382091:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.382097:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:1.0:1787697897.382102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697897/real 1787697897] req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697897.382102:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697897.382124:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697897.382128:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697897.382130:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697897.382133:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697897.382141:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697897.382148:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697897.382149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382160:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697897.382165:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787697897.382169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697897.382175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697897.382182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697897.382189:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382193:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697897.382204:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697897.382210:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5808) 00000100:00100000:1.0:1787697897.382213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697897.382225:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382231:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697897.382240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697897.382245:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382255:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697897.382259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697897/real 1787697897] req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382272:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697897.382276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697897.382278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697897.382281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697897.382289:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126271104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697897.382294:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697897.382298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126271232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697897.382303:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697897.382308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697897.382322:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382327:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697897.382333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697897.382337:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697897.382346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697897/real 1787697897] req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697897.382364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697897.382366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697897.382368:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697897.382376:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697897.382387:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382396:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697897.382402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697897.382409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697922 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382449:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697897.382457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697897/real 1787697897] req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697897.382481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697897.382489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697897.382492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697897.382497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697897.382505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697897.382511:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126271360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697897.382519:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697897.382525:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126271488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697922 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697897.382533:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697897.893568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697897.893568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697897.893575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697898.021610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697898.917520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697898.917520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697898.917547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697899.045577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697899.301520:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300472687, at 4300472704 00000100:00000040:3.0:1787697899.301520:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300472687, at 4300472704 00000100:00100000:1.0:1787697899.941528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697899.941528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697899.941629:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697900.069597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697900.965532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697900.965533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697900.965553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697901.093624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697901.989521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697901.989525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697901.989525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697902.117635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697902.501578:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697902.501591:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697902.501596:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697902.501610:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5803 00000100:00080000:3.0:1787697902.501625:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697902.501629:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697902.501635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.501639:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697902.501644:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697902.501649:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697902.501652:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.501655:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697902.501693:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697902.501709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697902.501740:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697902.501756:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697902.501778:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697902.501781:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697902.501783:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697902.501789:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5803 00000100:00080000:3.0:1787697902.501793:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697902.501795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697902.501797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.501799:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697902.501801:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697902.501804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697902.501806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.501808:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697902.501812:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697902.501823:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697902.501827:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697902.501831:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697902.501848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126271616:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787697902.501849:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697902.501862:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.501881:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.501899:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697902.501915:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697902.501919:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697902.501921:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697902.501927:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5803 00000100:00080000:3.0:1787697902.501932:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697902.501935:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697902.501939:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697902.501940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.501944:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697902.501947:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697902.501952:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697902.501955:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.501958:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787697902.501962:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697902.501964:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697902.501975:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697902.501987:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697902.501999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697902/real 1787697902] req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697902.502003:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697902.502019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697902.502038:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502043:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697902.502065:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697902.502066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697902.502073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787697902.502074:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:1.0:1787697902.502078:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697902.502079:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697902.502089:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5803 00000100:00100000:1.0:1787697902.502091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126271616:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697902.502098:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697902.502100:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126271616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697902.502110:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787697902.502111:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697902.502116:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.502119:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697902.502122:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697902.502125:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697902.502129:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697902.502132:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697902.502139:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:1.0:1787697902.502142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271744:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697902.502146:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697902.502153:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697902.502160:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697902.502166:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502173:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697902.502174:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697902.502186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697902.502190:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502194:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502217:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787697902.502219:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697902.502224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697902/real 1787697902] req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697902.502225:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697902.502230:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697902.502238:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5803 00000100:00100000:1.0:1787697902.502241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697902.502246:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697902.502249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697902.502254:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697902.502257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787697902.502262:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697902.502263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697902.502269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697902.502270:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697902.502275:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787697902.502280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787697902.502284:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697902.502288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697902.502293:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697902.502298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271872:192.168.201.136@tcp:38:kworker.0 00000020:00000040:3.0:1787697902.502299:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787697902.502309:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697902.502312:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697902.502322:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787697902.502323:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697902.502329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697902.502331:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697902.502336:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787697902.502347:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697902.502351:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697902/real 1787697902] req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697902.502377:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697902.502403:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5813) 00000100:00080000:1.0:1787697902.502411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697902.502413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697902.502415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126271872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697902.502461:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126271744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697902.502466:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697902.502471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126271872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697902.502475:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697902.502480:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697902.502494:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502499:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697902.502505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697902.502509:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502515:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697902.502518:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697902/real 1787697902] req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697902.502553:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697902.502556:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697902.502560:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697902.502573:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502582:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697902.502593:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502597:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697902.502605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697902.502610:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697927 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502616:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697902.502619:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697902/real 1787697902] req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697902.502631:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697902.502635:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697902.502638:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697902.502640:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697902.502645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697902.502648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126272000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697902.502652:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697902.502655:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126272128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697927 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697902.502659:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697903.013514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697903.013514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697903.013552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697903.141568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697904.037517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697904.037517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697904.037517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697904.165594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697904.421512:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300477807, at 4300477824 00000100:00000040:3.0:1787697904.421512:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300477807, at 4300477824 00000100:00100000:2.0:1787697905.061536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697905.061536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697905.061655:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697905.189606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697906.085516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697906.085522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697906.085523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697906.213656:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697907.109561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697907.109561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697907.109565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697907.237576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697907.621539:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697907.621548:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697907.621552:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697907.621568:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5808 00000100:00080000:3.0:1787697907.621579:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697907.621583:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.621587:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621590:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697907.621595:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697907.621598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.621601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621603:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697907.621606:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621634:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697907.621657:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697907.621671:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697907.621693:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697907.621696:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697907.621698:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697907.621702:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5808 00000100:00080000:3.0:1787697907.621706:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697907.621708:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.621712:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621718:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697907.621722:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697907.621726:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.621729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621731:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697907.621734:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697907.621742:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697907.621743:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697907.621752:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697907.621763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126272256:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787697907.621766:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697907.621777:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697907.621787:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697907.621794:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697907.621799:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697907.621803:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697907.621812:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697907.621813:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5808 00000100:00080000:3.0:1787697907.621820:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697907.621824:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.621826:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621829:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697907.621831:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697907.621834:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.621837:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621839:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697907.621842:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697907.621847:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697907.621851:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697907.621852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697907.621865:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697907.621872:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697907.621881:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697907.621891:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697907.621894:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697907.621896:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697907.621900:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5808 00000100:00080000:3.0:1787697907.621904:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697907.621908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697907/real 1787697907] req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697907.621910:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.621917:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.621926:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697907.621929:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787697907.621930:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.621958:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697907.621966:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697907.621968:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787697907.621973:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697907.621975:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:3.0:1787697907.621979:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787697907.621983:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697907.621986:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697907.621996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697907.621998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126272256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697907.622005:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126272256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697907.622010:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787697907.622019:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697907.622026:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697907.622038:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622041:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272384:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697907.622066:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697907.622070:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697907.622072:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697907.622086:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5808 00000100:00000040:1.0:1787697907.622088:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697907.622093:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697907.622099:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697907.622103:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697907.622107:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697907.622110:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697907.622113:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697907.622116:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697907.622121:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697907.622122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697907.622125:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697907.622130:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697907.622135:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697907.622136:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697907.622144:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:3.0:1787697907.622148:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697907.622151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697907/real 1787697907] req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697907.622161:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697907.622165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697907.622188:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697907.622205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787697907.622210:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5818) 00000100:00080000:1.0:1787697907.622211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697907.622219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697907.622235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697907.622252:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622257:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697907.622265:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697907.622270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622279:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697907.622282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697907/real 1787697907] req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697907.622298:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697907.622300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697907.622302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697907.622309:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126272384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697907.622313:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697907.622317:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126272512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697907.622321:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697907.622326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697907.622339:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622343:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697907.622348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697907.622352:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622357:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697907.622360:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697907/real 1787697907] req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697907.622375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697907.622377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697907.622379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697907.622386:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697907.622396:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622403:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697907.622408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697907.622411:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697932 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622416:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697907.622442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697907/real 1787697907] req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697907.622454:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697907.622458:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697907.622459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697907.622461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697907.622465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126272768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697907.622472:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126272640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697907.622477:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697907.622480:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126272768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697932 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697907.622484:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697908.133499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697908.133499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697908.133524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697908.261590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697909.157557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697909.157557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697909.157559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697909.285585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697909.541525:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300482927, at 4300482944 00000100:00000040:3.0:1787697909.541525:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300482927, at 4300482944 00000100:00100000:0.0:1787697910.181524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697910.181524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697910.181524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697910.309595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697911.205525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697911.205525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697911.205525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697911.333591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697912.229530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697912.229534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697912.229536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697912.357589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697912.741554:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697912.741567:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697912.741572:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697912.741585:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5813 00000100:00080000:3.0:1787697912.741598:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697912.741602:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741606:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741610:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697912.741616:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697912.741619:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741625:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697912.741629:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741646:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697912.741676:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697912.741692:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697912.741721:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697912.741741:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697912.741743:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697912.741747:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5813 00000100:00080000:3.0:1787697912.741751:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697912.741754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741758:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741760:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697912.741762:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697912.741765:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741768:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741770:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697912.741772:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697912.741781:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697912.741786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697912.741787:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697912.741798:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697912.741810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126272896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697912.741810:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697912.741827:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697912.741829:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697912.741831:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697912.741833:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5813 00000100:00080000:3.0:1787697912.741836:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697912.741838:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741839:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741845:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787697912.741870:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697912.741870:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697912.741876:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741886:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697912.741889:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697912.741894:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697912.741895:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741899:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697912.741909:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697912.741916:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697912.741922:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697912.741924:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697912.741925:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697912.741927:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5813 00000100:00080000:3.0:1787697912.741932:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697912.741934:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741936:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741937:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697912.741939:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697912.741940:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.741942:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.741943:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697912.741947:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:1.0:1787697912.741948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787697912.741951:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697912.741959:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697912.741975:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697912.741976:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697912.741995:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697912.742002:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697912.742004:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697912.742005:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697912.742007:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5813 00000100:00080000:3.0:1787697912.742010:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697912.742012:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.742014:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.742016:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697912.742017:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697912.742019:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697912.742021:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697912.742022:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697912.742024:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697912.742027:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697912.742032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697912/real 1787697912] req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697912.742032:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697912.742041:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697912.742052:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697912.742059:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5823) 00000100:00100000:1.0:1787697912.742063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697912.742099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697912.742104:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697912.742108:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126272896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697912.742124:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126272896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697912.742136:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697912.742155:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697912.742179:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742187:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697912.742200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697912.742207:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742217:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697912.742223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697912/real 1787697912] req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742234:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697912.742250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697912.742253:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697912.742257:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697912.742270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697912.742291:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742298:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697912.742308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697912.742314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742323:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697912.742328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697912/real 1787697912] req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697912.742354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697912.742357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697912.742361:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697912.742380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697912.742402:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742410:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697912.742452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697912.742459:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742469:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697912.742475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697912/real 1787697912] req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742494:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697912.742502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697912.742505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697912.742509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697912.742531:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697912.742553:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742560:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697912.742571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697912.742578:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697937 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742586:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697912.742592:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697912/real 1787697912] req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697912.742612:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697912.742619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697912.742622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697912.742625:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697912.742632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697912.742638:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126273024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697912.742652:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697912.742658:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126273152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697912.742665:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697912.742670:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126273280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697912.742677:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697912.742682:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126273408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697937 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697912.742688:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697913.253553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697913.253553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697913.253560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697913.381629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697914.277546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697914.277546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697914.277552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697914.405582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697914.661526:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300488047, at 4300488064 00000100:00000040:3.0:1787697914.661573:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300488047, at 4300488064 00000100:00100000:0.0:1787697915.301482:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697915.301488:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697915.301712:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697915.429586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697916.325512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697916.325512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697916.325512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697916.453560:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697917.349543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697917.349543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697917.349544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697917.477640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697917.861584:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697917.861593:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697917.861611:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697917.861621:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5818 00000100:00080000:0.0:1787697917.861631:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697917.861634:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861642:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697917.861646:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697917.861650:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861652:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861656:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697917.861659:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861669:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697917.861693:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697917.861708:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697917.861736:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697917.861739:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697917.861741:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697917.861745:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5818 00000100:00080000:0.0:1787697917.861749:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697917.861751:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861754:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861756:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697917.861759:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697917.861785:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861788:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861796:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697917.861799:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697917.861808:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697917.861814:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697917.861816:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697917.861828:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697917.861837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126273536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697917.861844:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697917.861855:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697917.861858:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697917.861861:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697917.861865:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5818 00000100:00080000:0.0:1787697917.861869:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697917.861871:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697917.861875:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861882:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861887:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697917.861891:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697917.861893:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861897:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861901:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861903:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697917.861906:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697917.861910:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697917.861914:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697917.861923:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697917.861934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861948:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.861956:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697917.861960:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697917.861967:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697917.861970:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697917.861976:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5818 00000100:00080000:0.0:1787697917.861980:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697917.861990:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.861993:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.861995:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697917.861997:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697917.862003:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.862010:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697917.862011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697917/real 1787697917] req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697917.862013:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697917.862019:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697917.862028:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697917.862035:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697917.862039:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697917.862047:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697917.862057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697917.862063:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697917.862077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697917.862079:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697917.862085:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697917.862085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787697917.862090:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697917.862095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697917.862104:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5818 00000100:00080000:0.0:1787697917.862111:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697917.862114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126273536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697917.862118:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.862124:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.862127:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787697917.862129:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126273536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697917.862131:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697917.862138:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697917.862145:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697917.862148:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000020:00000040:1.0:1787697917.862149:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787697917.862152:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697917.862158:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697917.862162:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697917.862172:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697917.862173:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697917.862209:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697917.862226:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5828) 00000100:00000040:1.0:1787697917.862232:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862242:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697917.862256:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697917.862264:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862274:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697917.862279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697917/real 1787697917] req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697917.862305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697917.862308:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697917.862312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697917.862325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697917.862341:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862345:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697917.862353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697917.862358:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862367:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697917.862370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697917/real 1787697917] req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697917.862386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697917.862388:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697917.862389:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697917.862397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697917.862407:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862411:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697917.862416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697917.862453:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862459:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697917.862463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697917/real 1787697917] req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697917.862478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697917.862480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697917.862482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862486:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126273920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697917.862489:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126273664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697917.862494:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697917.862497:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126273792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697917.862501:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697917.862504:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126273920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697917.862508:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697917.862513:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697917.862525:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862529:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697917.862535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697917.862538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697942 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862544:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697917.862546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697917/real 1787697917] req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862553:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697917.862557:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697917.862561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697917.862562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697917.862564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697917.862569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697917.862572:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126274048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697942 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697917.862575:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697918.373564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697918.373566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697918.373567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697918.501597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697919.397550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697919.397550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697919.397556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697919.525566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697919.781533:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300493167, at 4300493184 00000100:00000040:3.0:1787697919.781533:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300493167, at 4300493184 00000100:00100000:0.0:1787697920.421567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697920.421574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697920.421574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697920.549625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697921.445571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697921.445571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697921.445576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697921.573645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697922.469564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697922.469564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697922.469586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697922.597612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697922.981563:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697922.981572:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697922.981576:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697922.981587:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5823 00000100:00080000:3.0:1787697922.981597:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697922.981600:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981603:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981606:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697922.981609:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697922.981612:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981615:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981618:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697922.981622:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981637:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697922.981660:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697922.981685:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697922.981704:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697922.981725:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697922.981727:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697922.981733:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5823 00000100:00080000:3.0:1787697922.981737:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697922.981740:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981743:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981745:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697922.981747:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697922.981749:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981751:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981753:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697922.981755:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697922.981771:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697922.981776:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697922.981776:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697922.981790:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697922.981799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126274176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787697922.981800:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697922.981814:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697922.981817:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697922.981819:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697922.981822:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5823 00000100:00080000:3.0:1787697922.981824:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697922.981828:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981829:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981836:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697922.981838:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697922.981842:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697922.981845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981858:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981868:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697922.981880:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697922.981888:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787697922.981892:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697922.981898:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697922.981909:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697922.981916:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697922.981922:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697922.981924:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697922.981925:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697922.981927:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5823 00000100:00080000:3.0:1787697922.981930:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697922.981932:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981935:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981937:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697922.981938:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697922.981940:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.981943:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.981945:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697922.981947:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697922.981951:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697922.981960:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697922.981970:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697922.981981:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697922.981982:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697922.982022:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697922.982026:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697922.982027:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697922.982038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274304:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697922.982039:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5823 00000100:00080000:3.0:1787697922.982047:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697922.982053:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.982055:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.982056:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697922.982058:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697922.982060:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697922.982062:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697922.982064:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697922.982073:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697922.982074:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697922.982082:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697922.982089:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697922.982090:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697922.982103:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697922.982110:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697922.982111:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787697922.982127:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5834) 00000100:00000040:1.0:1787697922.982128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697922.982145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697922.982161:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697922.982176:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697922.982187:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787697922.982195:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697922.982201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697922.982210:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697922.982214:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697922.982223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787697922.982229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697922.982237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697922.982249:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697922.982255:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697922.982266:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787697923.493561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697923.493561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697923.493562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697923.621707:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697924.517526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697924.517526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697924.517526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697924.645594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697924.901529:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300498287, at 4300498304 00000100:00000040:3.0:1787697924.901529:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300498287, at 4300498304 00000100:00100000:0.0:1787697925.541509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697925.541509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697925.541509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697925.669567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787697926.053665:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787697926.053709:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787697926.053739:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697926.053782:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697926.053798:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697926.053806:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697926.053815:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697926.053823:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697926.053833:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787697926.053841:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787697926.053854:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787697926.053857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697922/real 1787697926] req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697926.053864:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697947 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.053887:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697926.053901:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697926.053912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697926.053917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697926.053921:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.053929:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274560:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697926.053936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697922/real 1787697926] req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.053949:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697926.053958:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697926.053963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697926.053966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697926.053968:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.053973:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274688:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697926.053978:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697922/real 1787697926] req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.054000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697926.054008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697926.054014:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697926.054017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697926.054020:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.054027:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274304:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697926.054032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697922/real 1787697926] req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.054042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697926.054049:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697926.054056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697926.054059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697926.054062:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.054068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274432:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697926.054075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697922/real 1787697926] req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.054084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697926.054092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697926.054098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697926.054100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697926.054104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697926.054110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126274176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697926.054116:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126274560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697926.054127:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697926.054136:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126274688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697926.054141:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697926.054146:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126274304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697926.054151:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697926.054155:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126274432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697926.054161:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697926.054166:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126274176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697947 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697926.054172:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697926.565516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697926.565516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697926.565516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697926.693655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697927.589501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697927.589501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697927.589510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697927.717640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697928.101548:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697928.101561:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697928.101566:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697928.101578:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5829 00000100:00080000:3.0:1787697928.101601:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697928.101605:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.101610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.101614:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697928.101619:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697928.101623:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.101626:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.101629:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697928.101632:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697928.101647:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697928.101677:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697928.101694:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697928.101728:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697928.101733:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697928.101735:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697928.101759:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5829 00000100:00080000:3.0:1787697928.101763:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697928.101766:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.101770:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.101772:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697928.101776:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697928.101779:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.101782:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.101785:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787697928.101790:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697928.101790:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697928.101806:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697928.101814:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697928.101822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126274816:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787697928.101829:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697928.101847:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697928.101859:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697928.101864:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:1.0:1787697928.101865:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697928.101867:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697928.101877:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5829 00000100:00000040:1.0:1787697928.101882:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697928.101882:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697928.101889:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.101893:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.101895:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697928.101898:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697928.101902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.101912:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.101915:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697928.101918:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697928.101924:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697928.101930:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697928.101937:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787697928.101943:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697928.101955:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.101959:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697928.101978:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697928.101983:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697928.101986:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697928.101992:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5829 00000100:00080000:3.0:1787697928.101997:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697928.102000:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.102003:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.102006:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697928.102009:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787697928.102014:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697928.102015:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697928.102022:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.102028:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787697928.102034:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697928.102041:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787697928.102045:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697928.102049:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697928.102058:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697928.102066:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697928.102089:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697928.102090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697928.102100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787697928.102107:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697928.102110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697928.102112:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787697928.102116:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697928.102125:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5829 00000100:00100000:1.0:1787697928.102130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126274816:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787697928.102132:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697928.102140:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697928.102146:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126274816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697928.102146:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.102150:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787697928.102154:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697928.102161:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697928.102166:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787697928.102166:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697928.102170:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697928.102174:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697928.102180:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697928.102189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697928.102199:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697928.102201:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274944:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787697928.102214:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697928.102222:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697928.102233:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102240:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697928.102252:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5839) 00000800:00000100:1.0:1787697928.102259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697928.102268:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102286:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697928.102293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697928.102322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697928.102325:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697928.102329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126274944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697928.102341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697928.102363:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102370:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697928.102382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697928.102389:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102398:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697928.102403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102459:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697928.102467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697928.102470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697928.102473:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697928.102489:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697928.102508:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102514:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697928.102526:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697928.102533:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102540:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697928.102547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102557:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102565:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697928.102571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697928.102574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697928.102577:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697928.102605:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126274944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697928.102613:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697928.102620:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126275072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697928.102628:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697928.102634:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126275200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697928.102641:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697928.102649:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697928.102674:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102680:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697928.102704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697928.102711:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697953 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102721:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697928.102727:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697928/real 1787697928] req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697928.102747:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697928.102756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697928.102759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697928.102763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697928.102771:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697928.102777:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126275328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697953 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697928.102785:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697928.613567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697928.613592:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697928.613719:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697928.741612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697929.637500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697929.637531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697929.637556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697929.765646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697930.021599:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300503407, at 4300503424 00000100:00000040:3.0:1787697930.021846:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300503407, at 4300503424 00000100:00100000:1.0:1787697930.661515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697930.661521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697930.661531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697930.789571:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697931.685576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697931.685587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697931.685650:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697931.813605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697932.709492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697932.709551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697932.709551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697932.837647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787697933.221540:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697933.221576:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787697933.221582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697933.221595:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5834 00000100:00080000:3.0:1787697933.221608:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697933.221613:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.221619:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697933.221623:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787697933.221630:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787697933.221634:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.221637:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697933.221640:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787697933.221643:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787697933.221657:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787697933.221689:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697933.221706:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697933.221730:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697933.221734:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787697933.221736:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697933.221741:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5834 00000100:00080000:3.0:1787697933.221744:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697933.221748:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.221751:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697933.221754:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787697933.221757:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787697933.221767:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.221773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697933.221775:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697933.221776:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787697933.221782:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697933.221796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126275456:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787697933.221797:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697933.221805:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787697933.221819:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697933.221830:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697933.221833:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.221849:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787697933.221857:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697933.221861:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787697933.221863:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697933.221868:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5834 00000100:00080000:3.0:1787697933.221873:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787697933.221877:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.221880:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697933.221882:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787697933.221885:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787697933.221888:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.221893:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697933.221894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697933.221897:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787697933.221901:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787697933.221907:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787697933.221912:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697933.221918:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697933.221924:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787697933.221947:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697933.221956:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787697933.221957:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697933/real 1787697933] req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697933.221964:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787697933.221968:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787697933.221974:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5834 00000100:00080000:3.0:1787697933.221978:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697933.221979:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697933.221986:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.221992:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697933.221994:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697933.221996:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787697933.222001:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787697933.222005:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.222009:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697933.222012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787697933.222013:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697933.222018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787697933.222020:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697933.222025:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787697933.222028:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697933.222037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126275456:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787697933.222038:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697933.222046:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126275456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697933.222053:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787697933.222055:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787697933.222066:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697933.222072:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787697933.222074:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:1.0:1787697933.222077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787697933.222077:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787697933.222085:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275584:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787697933.222091:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5834 00000100:00080000:3.0:1787697933.222097:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697933.222101:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697933.222102:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787697933.222110:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787697933.222117:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787697933.222119:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787697933.222123:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787697933.222129:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787697933.222133:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787697933.222136:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697933.222141:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697933.222141:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787697933.222148:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697933.222155:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000008:00100000:3.0:1787697933.222157:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697933.222161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697933/real 1787697933] req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222169:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787697933.222177:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697933.222186:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697933.222188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697933.222197:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787697933.222197:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697933.222228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697933.222235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787697933.222235:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787697933.222270:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5844) 00000100:00000040:1.0:1787697933.222275:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222281:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697933.222287:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697933.222292:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222298:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697933.222302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697933/real 1787697933] req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222313:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697933.222317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697933.222319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697933.222321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222325:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697933.222328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222333:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697933.222340:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222345:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697933.222354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697933.222360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222369:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697933.222375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697933/real 1787697933] req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697933.222403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697933.222405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697933.222407:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697933.222415:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126275584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697933.222442:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697933.222448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126275712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697933.222455:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697933.222460:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126275840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697933.222467:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697933.222475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697933.222495:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222511:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697933.222520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697933.222525:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697958 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222531:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697933.222534:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697933/real 1787697933] req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697933.222546:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697933.222551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697933.222552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697933.222554:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697933.222558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126275968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697933.222561:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126275968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697958 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697933.222566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697933.733564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697933.733564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697933.733573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697933.861611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697934.757525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697934.757536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697934.757537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697934.885646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697935.141497:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300508527, at 4300508544 00000100:00000040:2.0:1787697935.141537:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300508527, at 4300508544 00000100:00100000:0.0:1787697935.781551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697935.781551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697935.781551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697935.909593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697936.805537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697936.805537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697936.805567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697936.933579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697937.829493:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697937.829494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697937.829496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697937.957569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787697938.341514:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697938.341524:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787697938.341528:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697938.341537:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5839 00000100:00080000:1.0:1787697938.341548:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697938.341551:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341555:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341557:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787697938.341561:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787697938.341563:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341565:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341567:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787697938.341569:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341580:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697938.341598:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697938.341609:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697938.341629:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697938.341632:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787697938.341633:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697938.341637:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5839 00000100:00080000:1.0:1787697938.341641:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697938.341643:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341645:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341647:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787697938.341650:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697938.341653:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341655:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341657:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787697938.341660:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697938.341668:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697938.341673:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697938.341676:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697938.341681:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697938.341691:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.341695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126276096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787697938.341699:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697938.341702:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787697938.341703:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697938.341707:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5839 00000100:00080000:1.0:1787697938.341710:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697938.341712:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341714:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341716:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787697938.341718:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697938.341720:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697938.341721:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341723:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341726:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787697938.341728:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697938.341728:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787697938.341739:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787697938.341743:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697938.341750:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697938.341759:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697938.341763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787697938.341767:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697938.341770:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787697938.341771:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697938.341775:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5839 00000100:00080000:1.0:1787697938.341778:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697938.341780:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697938.341781:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697938.341783:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341785:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787697938.341787:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697938.341790:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341792:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341794:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787697938.341796:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787697938.341800:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697938.341804:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697938.341809:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697938/real 1787697938] req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697938.341811:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697938.341824:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697938.341825:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697938.341833:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787697938.341836:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697938.341838:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697938.341840:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697938.341841:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5839 00000100:00080000:1.0:1787697938.341844:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697938.341847:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341849:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697938.341849:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787697938.341851:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:2.0:1787697938.341852:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787697938.341853:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787697938.341856:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697938.341856:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697938.341859:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697938.341861:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697938.341863:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697938.341865:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126276096:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787697938.341867:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787697938.341871:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697938.341872:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126276096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697938.341877:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787697938.341880:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697938.341885:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697938.341893:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5849) 00000100:00000040:2.0:1787697938.341894:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.341907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697938.341921:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.341928:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697938.341940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697938.341945:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.341953:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697938.341956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697938/real 1787697938] req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.341963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.341968:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697938.341973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697938.341974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697938.341976:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.341987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697938.341991:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.341996:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697938.342004:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342008:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697938.342014:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697938.342018:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342023:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697938.342026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697938/real 1787697938] req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697938.342041:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697938.342043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697938.342045:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697938.342052:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697938.342062:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342067:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697938.342072:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697938.342075:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342080:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697938.342082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697938/real 1787697938] req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697938.342097:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697938.342100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697938.342102:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697938.342110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342118:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697938.342128:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342134:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697938.342144:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697938.342150:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697963 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342159:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697938.342164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697938/real 1787697938] req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697938.342178:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697938.342183:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697938.342185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697938.342188:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697938.342192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697938.342196:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126276224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697938.342200:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697938.342204:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126276352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697938.342207:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697938.342210:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126276480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697938.342214:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697938.342217:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126276608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697963 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697938.342221:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697938.853497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697938.853497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697938.853501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697938.981573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697939.877518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697939.877518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697939.877526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697940.005605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697940.261490:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300513647, at 4300513664 00000100:00000040:3.0:1787697940.261490:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300513647, at 4300513664 00000100:00100000:3.0:1787697940.901478:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697940.901484:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697940.901484:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697941.029594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697941.925482:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697941.925493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697941.925495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697942.053590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697942.949529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697942.949529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697942.949529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697943.077605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697943.461530:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697943.461539:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697943.461543:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697943.461553:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5844 00000100:00080000:0.0:1787697943.461563:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697943.461566:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697943.461570:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461572:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697943.461576:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697943.461578:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697943.461579:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461581:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697943.461583:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461593:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697943.461613:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697943.461626:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697943.461646:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697943.461650:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697943.461651:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697943.461655:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5844 00000100:00080000:0.0:1787697943.461658:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697943.461670:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697943.461674:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697943.461680:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461684:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697943.461687:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697943.461689:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697943.461693:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126276736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697943.461695:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461699:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697943.461701:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697943.461707:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697943.461711:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697943.461718:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697943.461725:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697943.461730:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697943.461740:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697943.461750:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697943.461753:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697943.461755:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787697943.461760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697943.461761:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5844 00000100:00080000:0.0:1787697943.461767:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697943.461771:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697943.461775:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461778:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697943.461780:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697943.461784:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697943.461788:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697943.461792:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461796:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697943.461806:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697943.461810:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697943.461814:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697943.461824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697943/real 1787697943] req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697943.461825:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697943.461853:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697943.461858:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697943.461860:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697943.461861:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697943.461866:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697943.461867:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5844 00000100:00080000:0.0:1787697943.461872:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697943.461878:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697943.461881:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697943.461884:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461889:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697943.461892:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:2.0:1787697943.461894:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787697943.461897:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697943.461901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697943.461902:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461906:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787697943.461909:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697943.461911:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697943.461917:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697943.461920:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126276736:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697943.461922:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697943.461929:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126276736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697943.461931:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787697943.461945:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697943.461946:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697943.461956:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697943.461961:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:2.0:1787697943.461961:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697943.461964:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697943.461972:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5844 00000100:00080000:0.0:1787697943.461976:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697943.461976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697943.461982:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697943.461985:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.461987:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697943.461989:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697943.461992:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697943.461994:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697943.462001:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697943.462005:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787697943.462008:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697943.462011:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697943.462017:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697943.462023:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:2.0:1787697943.462023:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697943.462034:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697943.462037:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697943.462050:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:0.0:1787697943.462054:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462061:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697943/real 1787697943] req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697943.462074:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5854) 00000100:00100000:2.0:1787697943.462082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462090:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697943.462099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697943.462101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697943.462105:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462113:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697943.462118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697943.462139:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462146:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697943.462159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697943.462166:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462175:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697943.462180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697943/real 1787697943] req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462199:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697943.462207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697943.462211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697943.462214:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126276992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697943.462227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697943.462261:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462269:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697943.462281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697943.462288:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462297:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697943.462303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697943/real 1787697943] req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462319:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697943.462349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697943.462351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697943.462354:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697943.462362:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126276864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697943.462367:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697943.462371:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126276992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697943.462375:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697943.462378:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126277120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697943.462382:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697943.462388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697943.462407:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462414:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697943.462445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697943.462452:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697968 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462461:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697943.462467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697943/real 1787697943] req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697943.462487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697943.462491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697943.462493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697943.462495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697943.462499:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697943.462502:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126277248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697968 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697943.462506:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697943.973557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697943.973557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697943.973582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697944.101622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697944.997514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697944.997515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697944.997515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697945.125616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697945.381559:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300518767, at 4300518784 00000100:00000040:3.0:1787697945.381559:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300518767, at 4300518784 00000100:00100000:0.0:1787697946.021540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697946.021540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697946.021540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697946.149609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697947.045484:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697947.045490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697947.045507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697947.173598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697948.069529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697948.069529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697948.069529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697948.197568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697948.581520:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697948.581528:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697948.581531:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697948.581539:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5849 00000100:00080000:0.0:1787697948.581549:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697948.581551:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.581553:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581557:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697948.581561:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697948.581563:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.581564:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581566:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697948.581568:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581580:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697948.581599:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697948.581609:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697948.581636:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697948.581638:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697948.581640:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697948.581643:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5849 00000100:00080000:0.0:1787697948.581645:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697948.581646:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.581647:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581648:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697948.581670:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697948.581674:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.581677:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581680:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697948.581683:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787697948.581689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697948.581693:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697948.581699:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697948.581715:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787697948.581715:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126277376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697948.581734:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697948.581744:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697948.581747:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697948.581749:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697948.581751:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697948.581754:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5849 00000100:00080000:0.0:1787697948.581759:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697948.581763:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.581768:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697948.581770:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581772:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697948.581776:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697948.581780:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.581783:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581785:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697948.581788:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697948.581793:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697948.581798:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787697948.581805:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697948.581821:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697948.581823:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697948.581842:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697948.581867:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697948.581870:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697948.581872:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697948.581876:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5849 00000100:00080000:0.0:1787697948.581881:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697948.581885:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.581888:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581892:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:2.0:1787697948.581892:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697948/real 1787697948] req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697948.581897:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697948.581911:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787697948.581914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697948.581916:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.581920:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697948.581925:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697948.581932:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787697948.581937:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697948.581938:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787697948.581949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787697948.581951:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787697948.581957:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697948.581963:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.581967:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697948.581973:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697948.581978:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787697948.581983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126277376:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697948.581985:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697948.581991:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126277376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697948.581992:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5849 00000100:00080000:0.0:1787697948.582000:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697948.582005:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697948.582006:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697948.582010:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697948.582014:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697948.582016:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697948.582020:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697948.582025:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697948.582026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697948.582028:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697948.582035:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697948.582043:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787697948.582046:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277504:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787697948.582051:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697948.582061:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697948.582066:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697948.582081:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697948.582089:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5859) 00000100:00000040:2.0:1787697948.582095:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697948.582109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697948.582118:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582133:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787697948.582139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697948/real 1787697948] req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582148:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582155:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697948.582163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697948.582166:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697948.582170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697948.582183:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697948.582201:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582208:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697948.582218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697948.582225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582233:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697948.582251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697948/real 1787697948] req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582262:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697948.582276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697948.582279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697948.582283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697948.582295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697948.582313:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582319:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697948.582328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697948.582334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697948.582347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697948/real 1787697948] req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697948.582376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697948.582379:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697948.582382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697948.582394:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126277504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697948.582401:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697948.582407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126277632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697948.582413:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697948.582445:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126277760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697948.582452:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697948.582460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697948.582480:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582487:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697948.582496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697948.582502:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697973 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582510:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697948.582515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697948/real 1787697948] req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582524:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697948.582532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697948.582538:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697948.582541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697948.582544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697948.582551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126277888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697948.582555:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126277888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697973 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697948.582562:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697949.093517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697949.093517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697949.093517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697949.221594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697950.117521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787697950.117522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697950.117522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697950.245522:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697950.501519:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300523887, at 4300523904 00000100:00000040:3.0:1787697950.501519:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300523887, at 4300523904 00000100:00100000:1.0:1787697951.141532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697951.141532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697951.141561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697951.269613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697952.165516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697952.165516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697952.165516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697952.293594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697953.189532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697953.189532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697953.189616:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697953.317593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697953.701510:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697953.701520:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697953.701524:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697953.701534:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5854 00000100:00080000:0.0:1787697953.701545:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697953.701549:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697953.701553:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701572:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697953.701577:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697953.701581:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697953.701583:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701586:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697953.701589:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701599:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697953.701624:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697953.701638:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697953.701663:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697953.701667:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697953.701668:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697953.701673:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5854 00000100:00080000:0.0:1787697953.701678:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697953.701682:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697953.701685:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701687:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697953.701691:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697953.701697:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697953.701698:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697953.701700:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701705:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697953.701711:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697953.701724:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787697953.701724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126278016:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697953.701732:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697953.701755:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697953.701755:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.701780:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697953.701785:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697953.701796:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697953.701799:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697953.701802:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697953.701806:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5854 00000100:00080000:0.0:1787697953.701812:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787697953.701816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697953.701819:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697953.701823:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701826:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697953.701829:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697953.701832:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697953.701835:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697953.701839:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701844:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697953.701850:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697953.701856:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697953.701862:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697953.701872:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697953.701882:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.701890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697953/real 1787697953] req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697953.701893:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697953.701902:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697953.701911:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787697953.701915:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697953.701926:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5854 00000100:00000040:2.0:1787697953.701928:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697953.701933:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787697953.701936:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787697953.701940:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697953.701942:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697953.701949:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697953.701950:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697953.701956:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697953.701961:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787697953.701962:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126278016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697953.701970:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697953.701971:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126278016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697953.701978:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.701984:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697953.701991:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697953.701993:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:0.0:1787697953.701998:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697953.702004:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697953.702008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278144:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697953.702019:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697953.702030:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697953.702034:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702042:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697953.702052:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787697953.702055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697953.702059:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697953.702062:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787697953.702063:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697953.702071:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5854 00000100:00000040:2.0:1787697953.702075:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:0.0:1787697953.702078:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787697953.702082:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697953/real 1787697953] req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697953.702085:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697953.702093:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697953.702098:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:2.0:1787697953.702099:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697953.702103:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697953.702107:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697953.702115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787697953.702118:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787697953.702121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697953.702126:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697953.702130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697953.702131:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697953.702138:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787697953.702158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278144:192.168.201.136@tcp:38:kworker.0 00000008:00100000:0.0:1787697953.702161:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697953.702170:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697953.702170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278272:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787697953.702187:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697953.702195:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697953.702203:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702204:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697953.702214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697953.702217:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5864) 00000100:00000040:2.0:1787697953.702221:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702228:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787697953.702232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697953/real 1787697953] req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702240:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697953.702253:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697953.702256:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697953.702259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697953.702271:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126278144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697953.702279:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697953.702285:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126278272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697953.702292:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787697953.702299:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697953.702313:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702318:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697953.702325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697953.702329:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702335:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787697953.702338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697953/real 1787697953] req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697953.702353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697953.702355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697953.702357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697953.702365:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702369:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697953.702375:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702378:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787697953.702383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697953.702388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697978 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702395:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787697953.702400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697953/real 1787697953] req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697953.702436:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697953.702444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787697953.702447:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787697953.702451:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697953.702458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787697953.702464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126278400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697953.702472:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787697953.702478:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126278528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697978 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697953.702483:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697954.213487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697954.213488:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697954.213596:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697954.341570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697955.237520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697955.237521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697955.237521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697955.365584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697955.621507:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300529007, at 4300529024 00000100:00000040:3.0:1787697955.621507:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300529007, at 4300529024 00000100:00100000:1.0:1787697956.261490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697956.261504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697956.261548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697956.391168:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697957.285543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697957.285543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697957.285551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697957.413644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697958.309550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697958.309553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697958.309589:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697958.437614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697958.821549:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697958.821557:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697958.821560:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697958.821567:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5859 00000100:00080000:2.0:1787697958.821576:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697958.821578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821580:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821582:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697958.821586:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697958.821588:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821590:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821592:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697958.821594:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821604:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697958.821622:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697958.821633:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697958.821651:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697958.821653:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697958.821654:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697958.821657:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5859 00000100:00080000:2.0:1787697958.821659:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697958.821661:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821662:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821664:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697958.821665:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697958.821667:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821668:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821669:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697958.821671:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697958.821677:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697958.821680:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697958.821686:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697958.821696:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697958.821705:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697958.821709:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697958.821711:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697958.821716:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5859 00000100:00080000:2.0:1787697958.821721:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697958.821725:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821733:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697958.821739:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697958.821742:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697958.821747:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821767:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697958.821771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126278656:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787697958.821773:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697958.821786:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697958.821792:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697958.821798:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697958.821811:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697958.821818:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.821825:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697958.821828:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697958.821840:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697958.821848:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697958.821855:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5859 00000100:00080000:2.0:1787697958.821858:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697958.821860:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821861:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821862:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697958.821866:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697958.821869:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821876:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697958.821878:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821880:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697958.821883:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697958.821887:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697958.821892:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697958.821899:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697958.821906:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697958.821910:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697958.821912:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697958.821913:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697958.821914:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5859 00000100:00080000:2.0:1787697958.821916:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697958.821919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821921:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821923:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697958.821925:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697958.821928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697958.821930:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821932:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697958.821935:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697958.821939:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787697958.821944:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787697958.821948:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787697958.821962:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697958.821973:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697958.821980:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697958.822005:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5869) 00000100:00100000:1.0:1787697958.822037:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697958/real 1787697958] req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822096:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697958.822106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697958.822111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697958.822116:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126278656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697958.822132:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822142:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697958.822158:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822167:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697958.822180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697958.822188:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822199:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697958.822205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697958/real 1787697958] req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822225:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697958.822233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697958.822236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697958.822240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697958.822254:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126278656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697958.822265:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697958.822276:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126278784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697958.822284:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697958.822299:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697958.822323:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822330:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697958.822341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697958.822349:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822358:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697958.822364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697958/real 1787697958] req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822385:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697958.822393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697958.822396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697958.822400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126278912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697958.822414:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697958.822480:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822487:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697958.822499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697958.822506:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822516:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697958.822521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697958/real 1787697958] req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697958.822548:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697958.822551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697958.822555:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697958.822568:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697958.822589:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822601:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697958.822612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697958.822619:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697983 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822628:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697958.822633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697958/real 1787697958] req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697958.822653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697958.822660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697958.822663:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697958.822667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697958.822675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697958.822681:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126278912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697958.822689:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697958.822695:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126279040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697958.822702:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697958.822707:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126279168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697983 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697958.822715:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697959.333583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697959.333583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697959.333589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697959.461619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697960.357502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697960.357502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697960.357536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697960.485628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697960.741512:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300534127, at 4300534144 00000100:00000040:2.0:1787697960.741554:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300534127, at 4300534144 00000100:00100000:0.0:1787697961.381580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697961.381580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697961.381584:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697961.509606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697962.405505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697962.405505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697962.405541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697962.533562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697963.429529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697963.429529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697963.429529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697963.557574:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697963.941554:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697963.941561:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697963.941565:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697963.941574:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5864 00000100:00080000:0.0:1787697963.941583:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697963.941585:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941589:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941591:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697963.941595:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697963.941597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941599:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941617:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697963.941619:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941627:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697963.941651:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697963.941666:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697963.941686:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697963.941688:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697963.941690:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697963.941693:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5864 00000100:00080000:0.0:1787697963.941696:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697963.941698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941699:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941701:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697963.941703:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697963.941705:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941707:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941709:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697963.941711:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697963.941719:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697963.941722:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697963.941734:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697963.941735:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697963.941754:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.941763:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126279296:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697963.941777:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697963.941780:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697963.941781:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697963.941788:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5864 00000100:00080000:0.0:1787697963.941792:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697963.941793:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941800:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787697963.941800:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697963.941804:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697963.941809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941814:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941819:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697963.941823:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697963.941824:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787697963.941829:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697963.941834:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697963.941841:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697963.941848:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697963.941854:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697963.941855:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697963.941856:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697963.941862:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5864 00000800:00000100:1.0:1787697963.941864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697963.941866:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697963.941873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941875:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941877:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697963.941879:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787697963.941883:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697963.941886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941892:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941896:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697963.941901:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697963.941906:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697963.941908:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697963.941916:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697963.941923:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.941929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697963/real 1787697963] req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697963.941931:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697963.941937:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697963.941939:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697963.941946:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5864 00000100:00080000:0.0:1787697963.941951:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697963.941953:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697963.941955:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697963.941956:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697963.941959:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787697963.941960:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697963.941964:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697963.941990:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697963.941999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787697963.942000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787697963.942006:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787697963.942007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697963.942012:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697963.942018:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697963.942028:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697963.942030:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126279296:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787697963.942036:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697963.942040:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126279296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697963.942051:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787697963.942052:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697963.942066:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942072:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697963.942088:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5875) 00000100:00100000:1.0:1787697963.942089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697963.942105:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942112:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697963.942123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697963.942129:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942138:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697963.942144:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697963/real 1787697963] req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942154:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697963.942170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697963.942173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697963.942177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697963.942189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697963.942205:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942212:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697963.942220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697963.942225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942233:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697963.942238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697963/real 1787697963] req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942255:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697963.942262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697963.942265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697963.942268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697963.942286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697963.942302:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942308:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697963.942316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697963.942321:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942329:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697963.942334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697963/real 1787697963] req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697963.942357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697963.942359:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697963.942362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697963.942374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126279424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697963.942381:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697963.942386:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126279552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697963.942392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697963.942397:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126279680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697963.942404:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697963.942410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942437:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697963.942448:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942455:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697963.942465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697963.942471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697988 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942479:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697963.942484:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697963/real 1787697963] req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697963.942502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697963.942509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697963.942512:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697963.942516:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697963.942522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126279808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697963.942527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126279808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697988 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697963.942535:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697964.453505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697964.453505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697964.453507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697964.581609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697965.477534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697965.477549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697965.477549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697965.605612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697965.861548:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300539247, at 4300539264 00000100:00000040:2.0:1787697965.861656:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300539247, at 4300539264 00000100:00100000:0.0:1787697966.501536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697966.501536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697966.501536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697966.629614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697967.525555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697967.525555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697967.525559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697967.653560:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697968.549516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697968.549516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697968.549798:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697968.677605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697969.061535:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697969.061545:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697969.061549:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697969.061559:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5870 00000100:00080000:0.0:1787697969.061570:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697969.061573:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061577:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061581:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697969.061586:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697969.061589:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061592:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061594:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697969.061597:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061608:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697969.061646:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697969.061661:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.061682:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697969.061685:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697969.061686:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697969.061691:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5870 00000100:00080000:0.0:1787697969.061695:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697969.061698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061701:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061703:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697969.061705:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697969.061708:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061711:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061713:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697969.061715:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697969.061723:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787697969.061723:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697969.061728:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787697969.061736:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126279936:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787697969.061737:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697969.061745:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.061752:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697969.061755:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697969.061762:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697969.061765:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5870 00000100:00000040:1.0:1787697969.061768:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.061769:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697969.061772:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061775:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697969.061776:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061777:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697969.061779:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697969.061781:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061785:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697969.061787:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697969.061791:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697969.061794:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697969.061801:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697969.061806:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061810:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.061816:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697969.061819:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697969.061821:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697969.061824:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5870 00000100:00000040:1.0:1787697969.061825:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.061827:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697969.061830:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061833:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061834:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697969.061836:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697969.061843:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061846:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061848:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697969.061850:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697969.061853:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697969.061854:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697969.061858:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697969.061867:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697969.061876:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.061883:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.061903:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697969.061904:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697969.061906:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697969.061908:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697969.061911:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5870 00000100:00080000:1.0:1787697969.061912:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697969.061914:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787697969.061915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697969.061917:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697969.061919:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697969.061919:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697969.061921:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697969.061924:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697969.061926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697969.061927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126279936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697969.061929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697969.061931:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697969.061933:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697969.061934:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126279936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697969.061938:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697969.061941:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787697969.061943:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697969.061949:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697969.061957:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697969.061964:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.061965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697969.061974:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5880) 00000100:00000040:1.0:1787697969.061978:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.061984:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697969.062005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697969.062012:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062023:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697969.062029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062040:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062049:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697969.062056:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697969.062059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697969.062070:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697969.062083:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697969.062103:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062110:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697969.062122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697969.062127:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062141:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697969.062147:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697969.062179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697969.062182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697969.062186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697969.062194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697969.062208:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062213:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697969.062221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697969.062225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062231:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697969.062234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062248:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697969.062252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697969.062254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697969.062256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697969.062263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126280064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697969.062267:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697969.062271:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126280192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697969.062274:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697969.062277:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126280320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697969.062281:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697969.062286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062291:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697969.062298:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062302:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697969.062308:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697969.062311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697994 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062316:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697969.062319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697969/real 1787697969] req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697969.062330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697969.062333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697969.062335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697969.062337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697969.062341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697969.062343:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126280448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697994 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697969.062347:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697969.573556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697969.573556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697969.573563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697969.701571:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787697970.597552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697970.597577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697970.597954:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697970.725599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697970.981553:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300544367, at 4300544384 00000100:00000040:2.0:1787697970.981669:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300544367, at 4300544384 00000100:00100000:2.0:1787697971.621535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697971.621535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697971.621669:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697971.749635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697972.645544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697972.645544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697972.645568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697972.773607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697973.669513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697973.669513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697973.669513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697973.797636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697974.181555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697974.181569:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697974.181574:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697974.181587:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5875 00000100:00080000:0.0:1787697974.181598:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697974.181602:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181607:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181611:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697974.181617:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697974.181621:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181624:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181628:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697974.181631:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181655:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697974.181683:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697974.181695:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697974.181734:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697974.181737:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697974.181739:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697974.181743:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5875 00000100:00080000:0.0:1787697974.181745:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697974.181748:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181749:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181751:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697974.181753:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697974.181756:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181758:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181760:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697974.181762:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697974.181771:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697974.181774:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697974.181781:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697974.181790:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697974.181797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126280576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697974.181812:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697974.181824:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697974.181826:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697974.181828:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697974.181831:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5875 00000100:00080000:0.0:1787697974.181833:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697974.181840:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181842:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181843:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697974.181845:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787697974.181849:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697974.181850:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181856:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181861:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697974.181866:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697974.181869:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787697974.181875:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697974.181880:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697974.181888:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697974.181895:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697974.181900:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697974.181902:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697974.181904:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697974.181907:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5875 00000100:00080000:0.0:1787697974.181910:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697974.181911:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181913:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181915:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697974.181917:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:1.0:1787697974.181920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181921:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181924:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181925:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697974.181932:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697974.181937:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697974.181940:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697974.181943:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697974.181957:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697974.181965:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697974.181970:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697974.181971:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697974.181972:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697974.181974:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5875 00000100:00080000:0.0:1787697974.181978:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697974.181981:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.181984:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.181987:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697974.181989:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787697974.181991:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697974/real 1787697974] req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697974.181997:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697974.182004:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697974.182007:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697974.182010:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697974.182012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697974.182018:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697974.182022:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697974.182026:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697974.182031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697974.182035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787697974.182038:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697974.182042:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697974.182058:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126280576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697974.182068:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126280576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697974.182095:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697974.182098:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5885) 00000100:00000040:1.0:1787697974.182114:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182120:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697974.182130:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182135:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697974.182142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697974.182147:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182154:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697974.182157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697974/real 1787697974] req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697974.182174:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697974.182175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697974.182177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182186:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697974.182189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697974.182200:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182204:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697974.182209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697974.182212:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182218:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697974.182221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697974/real 1787697974] req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697974.182236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697974.182238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697974.182240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697974.182248:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697974.182260:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182264:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697974.182268:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697974.182271:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182277:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697974.182280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697974/real 1787697974] req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182291:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697974.182295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697974.182297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697974.182299:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126280960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697974.182305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697974.182317:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182321:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697974.182326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697974.182329:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787697999 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182334:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697974.182337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697974/real 1787697974] req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182343:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697974.182348:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697974.182352:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697974.182353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697974.182355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697974.182359:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697974.182361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126280704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697974.182366:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697974.182370:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126280832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697974.182375:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697974.182378:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126280960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697974.182382:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697974.182384:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126281088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787697999 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697974.182388:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697974.693539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697974.693540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697974.693540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697974.821565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697975.717542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697975.717542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697975.717543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697975.845615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697976.101546:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300549487, at 4300549504 00000100:00000040:3.0:1787697976.101546:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300549487, at 4300549504 00000100:00100000:3.0:1787697976.741553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697976.741562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697976.741568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697976.869631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697977.765538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697977.765570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697977.765572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697977.893620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697978.789561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697978.789563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697978.789564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697978.917615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697979.301563:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697979.301576:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697979.301580:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697979.301593:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5880 00000100:00080000:0.0:1787697979.301606:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697979.301609:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301612:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301615:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697979.301620:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697979.301622:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301624:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301627:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697979.301630:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301650:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697979.301681:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697979.301697:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697979.301751:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697979.301755:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697979.301774:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697979.301781:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5880 00000100:00080000:0.0:1787697979.301786:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787697979.301791:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697979.301792:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301800:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301805:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697979.301808:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00100000:1.0:1787697979.301810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126281216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697979.301814:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301817:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301818:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697979.301820:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697979.301829:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697979.301833:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697979.301843:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697979.301849:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697979.301856:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.301867:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697979.301868:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697979.301871:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697979.301873:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697979.301876:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5880 00000100:00080000:0.0:1787697979.301879:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697979.301881:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301883:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301885:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697979.301887:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697979.301895:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301897:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301899:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697979.301901:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697979.301904:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697979.301907:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697979.301916:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787697979.301917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301925:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697979.301932:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697979.301934:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697979.301936:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697979.301939:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697979.301941:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5880 00000100:00080000:0.0:1787697979.301947:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697979.301951:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301953:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301954:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697979.301956:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697979.301958:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.301960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.301961:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697979.301963:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697979.301966:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697979.301969:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697979.301974:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697979.301984:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697979/real 1787697979] req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697979.301989:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697979.302005:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697979.302008:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697979.302009:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697979.302012:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5880 00000100:00080000:0.0:1787697979.302014:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697979.302017:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697979.302018:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697979.302024:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.302027:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697979.302030:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697979.302034:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697979.302037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697979.302039:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697979.302044:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697979.302049:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697979.302054:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787697979.302054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787697979.302059:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787697979.302063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787697979.302067:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697979.302074:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697979.302079:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697979.302099:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5890) 00000100:00100000:1.0:1787697979.302099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126281216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697979.302110:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126281216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697979.302122:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787697979.302137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697979.302155:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302159:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697979.302169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697979.302173:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302184:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697979.302187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697979/real 1787697979] req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697979.302205:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697979.302206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697979.302208:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697979.302216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302221:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697979.302228:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302232:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697979.302237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697979.302240:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302246:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697979.302249:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697979/real 1787697979] req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697979.302265:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697979.302267:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697979.302269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302273:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697979.302275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697979.302286:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302290:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697979.302295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697979.302298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302303:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697979.302306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697979/real 1787697979] req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302316:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697979.302324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697979.302326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697979.302328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697979.302335:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697979.302345:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302349:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697979.302353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697979.302356:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698004 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302361:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697979.302364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697979/real 1787697979] req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697979.302375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697979.302380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697979.302381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697979.302383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697979.302387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697979.302389:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126281344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697979.302393:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697979.302396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126281472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697979.302400:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697979.302403:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126281600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697979.302406:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697979.302409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126281728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698004 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697979.302447:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697979.813528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697979.813528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697979.813529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697979.941613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697980.837527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697980.837527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697980.837534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697980.965622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697981.221498:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300554607, at 4300554624 00000100:00000040:3.0:1787697981.221498:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300554607, at 4300554624 00000100:00100000:0.0:1787697981.861514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697981.861514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697981.861514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697981.989611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697982.885529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697982.885530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697982.885530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697983.013568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697983.909491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697983.909491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697983.909491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697984.037619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697984.421536:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697984.421548:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697984.421553:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697984.421565:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5885 00000100:00080000:0.0:1787697984.421578:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697984.421581:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421585:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421589:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697984.421595:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697984.421599:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421602:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421606:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697984.421609:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421652:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697984.421684:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697984.421700:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697984.421723:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697984.421727:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697984.421729:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697984.421733:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5885 00000100:00080000:0.0:1787697984.421736:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697984.421739:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421759:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421762:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:1.0:1787697984.421764:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697984.421765:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787697984.421772:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421777:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421781:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697984.421786:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697984.421794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126281856:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787697984.421796:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697984.421801:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697984.421811:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697984.421822:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.421829:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697984.421835:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697984.421844:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:1.0:1787697984.421848:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697984.421849:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697984.421856:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5885 00000100:00080000:0.0:1787697984.421861:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697984.421865:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421868:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421871:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697984.421873:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697984.421876:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421878:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421881:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697984.421884:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697984.421895:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787697984.421899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421901:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697984.421911:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697984.421918:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697984.421925:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787697984.421926:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697984.421931:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697984.421935:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697984.421945:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5885 00000100:00080000:0.0:1787697984.421951:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697984.421954:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421957:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421959:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697984.421962:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697984.421965:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697984.421968:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.421971:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:1.0:1787697984.421974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697984/real 1787697984] req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697984.421976:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697984.421986:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697984.421992:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697984.422000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697984.422007:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697984.422017:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697984.422025:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697984.422034:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697984.422040:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697984.422047:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697984.422060:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697984.422065:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787697984.422068:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697984.422077:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5885 00000100:00100000:1.0:1787697984.422083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126281856:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787697984.422084:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697984.422092:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697984.422099:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126281856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697984.422100:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.422106:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787697984.422111:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697984.422118:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:1.0:1787697984.422119:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787697984.422123:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697984.422126:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697984.422129:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697984.422134:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787697984.422138:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697984.422143:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697984.422151:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697984.422160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697984.422167:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697984.422181:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5895) 00000100:00000040:1.0:1787697984.422183:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422194:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697984.422208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697984.422216:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422226:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697984.422232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697984/real 1787697984] req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697984.422260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697984.422263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697984.422267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126281984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697984.422286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697984.422307:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422314:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697984.422325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697984.422332:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422341:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697984.422347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697984/real 1787697984] req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697984.422373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697984.422376:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697984.422380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697984.422393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697984.422413:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422451:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697984.422463:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697984.422470:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422480:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697984.422486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697984/real 1787697984] req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422497:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422505:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697984.422513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697984.422516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697984.422520:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697984.422531:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126281984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697984.422539:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697984.422544:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126282112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697984.422550:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697984.422555:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126282240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697984.422561:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697984.422569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697984.422590:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422597:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697984.422609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697984.422617:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698009 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422641:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697984.422647:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697984/real 1787697984] req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697984.422667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697984.422680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697984.422683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697984.422687:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697984.422694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697984.422700:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126282368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698009 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697984.422708:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787697984.933516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697984.933516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697984.933533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697985.061668:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697985.957494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697985.957494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697985.957495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697986.085660:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787697986.341535:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300559727, at 4300559744 00000100:00000040:3.0:1787697986.341535:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300559727, at 4300559744 00000100:00100000:1.0:1787697986.981542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697986.981542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697986.981547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697987.109594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697988.005544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697988.005544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697988.005544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697988.133626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697989.029528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697989.029528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697989.029528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697989.157648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697989.541576:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697989.541585:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697989.541590:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697989.541601:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5890 00000100:00080000:2.0:1787697989.541613:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697989.541615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541626:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541629:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697989.541633:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697989.541636:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541638:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541640:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697989.541642:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541656:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697989.541682:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697989.541694:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697989.541711:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697989.541714:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697989.541715:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697989.541718:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5890 00000100:00080000:2.0:1787697989.541721:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697989.541722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541725:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697989.541728:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697989.541730:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541732:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541735:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697989.541737:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697989.541747:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697989.541752:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697989.541753:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697989.541762:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697989.541769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126282496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787697989.541773:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697989.541781:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697989.541785:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787697989.541787:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697989.541791:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5890 00000100:00080000:2.0:1787697989.541794:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697989.541797:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541800:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697989.541804:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697989.541804:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697989.541808:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697989.541817:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541824:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697989.541825:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541829:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697989.541833:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697989.541838:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697989.541843:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787697989.541855:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697989.541880:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697989.541889:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697989.541892:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697989.541894:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697989.541900:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5890 00000100:00080000:2.0:1787697989.541904:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697989.541908:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541914:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:1.0:1787697989.541915:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697989.541925:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787697989.541926:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697989.541933:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.541937:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697989.541940:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697989.541943:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697989.541949:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697989.541955:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697989.541959:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697989.541969:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787697989.541976:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697989.541986:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697989.541987:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787697989.541996:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697989.542016:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282752:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787697989.542019:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697989.542024:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697989.542029:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697989.542034:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697989.542036:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5890 00000100:00000040:1.0:1787697989.542043:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787697989.542048:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697989.542054:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697989.542057:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697989.542059:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000020:00000040:2.0:1787697989.542060:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697989.542065:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697989.542070:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697989.542074:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697989.542075:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787697989.542088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282880:192.168.201.136@tcp:8:kworker.0 00000020:00000040:2.0:1787697989.542091:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697989.542096:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697989.542101:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787697989.542106:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697989.542108:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697989.542117:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787697989.542123:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697989.542128:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:2.0:1787697989.542136:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697989.542151:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5900) 00000100:00000040:1.0:1787697989.542176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697989.542187:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697989.542203:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697989.542210:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697989.542221:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697990.053559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697990.053559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697990.053559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697990.181585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697991.077531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787697991.077532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697991.077557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697991.205649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697991.461479:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300564847, at 4300564864 00000100:00000040:2.0:1787697991.461495:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300564847, at 4300564864 00000100:00100000:1.0:1787697992.101535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787697992.101545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697992.101546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697992.229646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787697992.613826:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:2.0:1787697992.613888:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787697992.613911:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697992.613964:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697992.613996:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697992.614007:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697992.614018:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697992.614027:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697992.614038:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697992.614046:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787697992.614055:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787697992.614064:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698014 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697989/real 1787697992] req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697992.614193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697992.614207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697992.614212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697992.614218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283008:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697992.614235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697989/real 1787697992] req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697992.614258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697992.614267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697992.614271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697992.614275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614283:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282880:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787697992.614290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697989/real 1787697992] req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697992.614312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697992.614319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697992.614323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697992.614327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614336:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282624:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697992.614342:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697989/real 1787697992] req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697992.614370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697992.614378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697992.614382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697992.614385:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126282752:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787697992.614399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697989/real 1787697992] req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697992.614475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697992.614495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697992.614499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697992.614503:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697992.614509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126282496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787697992.614517:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126283008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697992.614536:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697992.614550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126282880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697992.614558:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697992.614564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126282624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697992.614571:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697992.614577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126282752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697992.614583:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697992.614589:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126282496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698014 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697992.614596:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787697993.125508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697993.125508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697993.125533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697993.253637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697994.149554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697994.149558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697994.149568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697994.277588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787697994.661516:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697994.661527:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787697994.661531:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697994.661542:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5895 00000100:00080000:2.0:1787697994.661554:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697994.661558:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697994.661561:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661564:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787697994.661568:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787697994.661570:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697994.661572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661574:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787697994.661577:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661585:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787697994.661608:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697994.661622:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697994.661646:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697994.661650:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787697994.661652:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697994.661657:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5895 00000100:00080000:2.0:1787697994.661661:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697994.661664:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697994.661668:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661670:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787697994.661672:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787697994.661675:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697994.661680:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697994.661683:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787697994.661684:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787697994.661687:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697994.661700:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697994.661708:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697994.661717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126283136:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787697994.661722:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697994.661733:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697994.661740:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697994.661744:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:0.0:1787697994.661749:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697994.661749:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697994.661756:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5895 00000100:00080000:2.0:1787697994.661760:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697994.661763:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697994.661766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661767:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787697994.661771:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787697994.661777:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697994.661780:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787697994.661784:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661787:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787697994.661803:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697994.661808:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787697994.661813:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:0.0:1787697994.661821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787697994.661823:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697994.661834:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.661840:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697994.661844:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787697994.661852:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787697994.661856:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697994.661860:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5895 00000100:00080000:2.0:1787697994.661865:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697994.661867:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787697994.661870:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661872:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787697994.661875:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787697994.661879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787697994.661883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697994/real 1787697994] req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697994.661884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.661889:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787697994.661895:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697994.661903:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697994.661909:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697994.661911:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697994.661921:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697994.661925:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697994.661945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787697994.661946:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697994.661952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:2.0:1787697994.661962:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787697994.661967:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787697994.661968:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787697994.661971:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787697994.661979:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5895 00000100:00100000:0.0:1787697994.661985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126283136:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787697994.661985:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787697994.661992:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697994.661998:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126283136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787697994.661999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.662004:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787697994.662008:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787697994.662014:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697994.662018:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787697994.662018:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787697994.662023:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787697994.662026:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787697994.662031:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787697994.662035:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787697994.662038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787697994.662047:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787697994.662065:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662073:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787697994.662080:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5905) 00000100:00000040:0.0:1787697994.662094:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662102:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787697994.662114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697994.662121:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662131:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787697994.662136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697994/real 1787697994] req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697994.662160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697994.662163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697994.662167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697994.662179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697994.662208:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662214:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787697994.662227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697994.662233:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662242:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787697994.662248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697994/real 1787697994] req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697994.662276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697994.662279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697994.662283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787697994.662295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787697994.662311:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662323:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787697994.662335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697994.662342:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662350:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787697994.662357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697994/real 1787697994] req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697994.662381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697994.662384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697994.662388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787697994.662400:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126283264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697994.662406:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787697994.662412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126283392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697994.662440:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787697994.662446:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126283520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697994.662470:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787697994.662480:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787697994.662497:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662503:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787697994.662515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697994.662522:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698019 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662529:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787697994.662535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697994/real 1787697994] req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697994.662551:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697994.662558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697994.662561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787697994.662564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697994.662583:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787697994.662601:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126283648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698019 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697994.662608:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787697995.173563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787697995.173564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697995.173568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787697995.301610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787697996.197574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697996.197575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787697996.197575:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697996.325554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787697996.581592:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300569967, at 4300569984 00000100:00000040:2.0:1787697996.582480:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300569967, at 4300569985 00000100:00100000:3.0:1787697997.221595:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697997.221596:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697997.221735:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697997.349627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787697998.245533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697998.245533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787697998.245533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697998.373649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787697999.269543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787697999.269544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787697999.269548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787697999.397632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787697999.781517:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697999.781530:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787697999.781534:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697999.781550:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5900 00000100:00080000:0.0:1787697999.781562:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697999.781565:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781568:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781572:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787697999.781576:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787697999.781579:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781580:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781583:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787697999.781585:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781600:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787697999.781625:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697999.781637:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697999.781660:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697999.781680:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787697999.781681:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697999.781685:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5900 00000100:00080000:0.0:1787697999.781689:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697999.781691:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781693:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781694:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787697999.781698:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787697999.781700:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697999.781702:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781708:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781712:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787697999.781717:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787697999.781724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126283776:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787697999.781727:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697999.781731:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787697999.781740:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697999.781752:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.781757:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697999.781764:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697999.781768:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787697999.781771:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787697999.781771:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787697999.781776:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5900 00000100:00080000:0.0:1787697999.781780:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697999.781781:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781789:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787697999.781791:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787697999.781793:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781795:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781797:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787697999.781802:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697999.781806:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787697999.781810:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787697999.781813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787697999.781820:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787697999.781829:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.781833:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697999.781841:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697999.781845:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787697999.781846:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787697999.781850:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5900 00000100:00080000:0.0:1787697999.781852:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787697999.781854:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781857:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781858:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787697999.781860:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787697999.781862:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781864:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781865:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787697999.781867:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787697999.781871:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697999.781872:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697999/real 1787697999] req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787697999.781882:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787697999.781891:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787697999.781892:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697999.781915:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.781918:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697999.781932:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787697999.781935:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787697999.781935:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787697999.781940:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787697999.781941:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787697999.781949:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5900 00000100:00000040:1.0:1787697999.781949:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787697999.781956:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787697999.781959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126283776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787697999.781964:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787697999.781968:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126283776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697999.781971:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781978:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787697999.781982:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781983:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787697999.781989:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787697999.781992:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787697999.781994:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787697999.781997:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787697999.781999:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787697999.782004:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787697999.782009:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283904:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787697999.782012:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697999.782022:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787697999.782027:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787697999.782034:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787697999.782044:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787697999.782045:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697999.782053:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787697999.782063:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5910) 00000100:00000040:1.0:1787697999.782065:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787697999.782069:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697999/real 1787697999] req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782084:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697999.782089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697999.782090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697999.782093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126283904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697999.782101:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697999.782114:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782118:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697999.782123:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697999.782126:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782131:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787697999.782134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697999/real 1787697999] req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782140:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697999.782149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697999.782150:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697999.782152:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787697999.782159:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697999.782171:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782176:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697999.782180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697999.782184:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782189:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787697999.782192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697999/real 1787697999] req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697999.782207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697999.782208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697999.782210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782214:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697999.782216:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126283904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697999.782220:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697999.782224:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126284032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697999.782228:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787697999.782231:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126284160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697999.782235:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787697999.782239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697999.782250:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782254:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787697999.782260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787697999.782263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698024 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782268:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787697999.782271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787697999/real 1787697999] req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787697999.782282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787697999.782286:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787697999.782287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787697999.782289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787697999.782293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787697999.782298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126284288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698024 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787697999.782302:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698000.293533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698000.293534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698000.293534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698000.421613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698001.317516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698001.317516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698001.317836:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698001.445632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698001.701552:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300575087, at 4300575104 00000100:00000040:3.0:1787698001.701558:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300575087, at 4300575104 00000100:00100000:0.0:1787698002.341521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698002.341521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698002.341571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698002.469570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698003.365498:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698003.365506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698003.365531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698003.493615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698004.389520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698004.389520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698004.389520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698004.517558:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698004.901520:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698004.901530:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698004.901533:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698004.901544:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5905 00000100:00080000:0.0:1787698004.901554:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698004.901557:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901561:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901564:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698004.901569:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698004.901572:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901575:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901577:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698004.901580:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901593:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698004.901615:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698004.901630:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698004.901655:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698004.901659:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698004.901661:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698004.901667:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5905 00000100:00080000:0.0:1787698004.901685:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698004.901690:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901693:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901696:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698004.901699:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698004.901702:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901705:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901709:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787698004.901712:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698004.901714:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698004.901726:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698004.901731:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126284416:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698004.901733:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698004.901748:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698004.901764:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698004.901774:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.901777:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698004.901789:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698004.901792:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698004.901794:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698004.901799:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5905 00000100:00080000:0.0:1787698004.901803:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698004.901807:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901809:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901812:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698004.901816:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:1.0:1787698004.901816:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901821:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901825:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901832:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787698004.901837:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698004.901838:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698004.901846:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698004.901852:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698004.901862:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698004.901875:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.901875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698004/real 1787698004] req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698004.901900:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698004.901904:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698004.901907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698004.901908:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698004.901916:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5905 00000100:00080000:0.0:1787698004.901923:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698004.901927:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698004.901929:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.901935:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901941:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698004.901944:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:1.0:1787698004.901947:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698004.901949:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698004.901956:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698004.901957:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.901962:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698004.901965:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698004.901967:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698004.901975:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698004.901980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126284416:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698004.901981:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698004.901993:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698004.902005:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126284416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698004.902027:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698004.902028:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787698004.902039:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698004.902043:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698004.902045:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698004.902047:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698004.902051:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5905 00000100:00080000:0.0:1787698004.902057:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698004.902062:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698004.902065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698004.902068:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698004.902073:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698004.902075:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698004.902078:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698004.902081:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698004.902084:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698004.902085:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698004.902090:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698004.902100:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698004.902100:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698004.902107:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:1.0:1787698004.902115:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698004.902120:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698004.902124:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698004.902143:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902146:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:0.0:1787698004.902156:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5916) 00000100:00100000:1.0:1787698004.902163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698004/real 1787698004] req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698004.902196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698004.902200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698004.902203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902210:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698004.902216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698004.902234:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902241:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698004.902257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698004.902264:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902272:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698004.902292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698004/real 1787698004] req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698004.902319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698004.902323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698004.902326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698004.902338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698004.902357:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902363:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698004.902373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698004.902386:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902395:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698004.902400:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698004/real 1787698004] req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902410:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902446:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698004.902454:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698004.902457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698004.902461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902468:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698004.902473:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126284544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698004.902480:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698004.902485:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126284672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698004.902492:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698004.902496:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126284800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698004.902502:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698004.902508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698004.902526:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902532:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698004.902542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698004.902549:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698029 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902557:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698004.902561:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698004/real 1787698004] req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698004.902584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698004.902590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698004.902593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698004.902604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698004.902611:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126284928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698004.902615:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126284928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698029 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698004.902622:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698005.413489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698005.413489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698005.413489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698005.541598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698006.437497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698006.437499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698006.437500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698006.565536:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698006.821507:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300580207, at 4300580224 00000100:00000040:3.0:1787698006.821507:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300580207, at 4300580224 00000100:00100000:0.0:1787698007.461534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698007.461534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698007.461535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698007.589565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698008.485512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698008.485512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698008.485513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698008.613550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698009.509500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698009.509502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698009.509503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698009.637588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698010.021527:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698010.021536:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698010.021539:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698010.021547:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5911 00000100:00080000:1.0:1787698010.021555:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698010.021558:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021561:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021564:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698010.021568:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698010.021570:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021573:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698010.021575:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021585:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698010.021604:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698010.021614:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698010.021631:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698010.021634:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698010.021635:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698010.021638:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5911 00000100:00080000:1.0:1787698010.021640:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698010.021642:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021643:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021645:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698010.021646:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698010.021648:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021650:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021651:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698010.021653:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698010.021659:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698010.021664:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698010.021674:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698010.021675:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698010.021691:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698010.021701:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698010.021704:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698010.021706:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698010.021709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126285056:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698010.021713:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5911 00000100:00080000:1.0:1787698010.021719:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698010.021722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021725:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021727:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698010.021730:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698010.021733:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021736:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698010.021740:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698010.021741:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698010.021749:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698010.021753:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787698010.021769:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698010.021777:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:0.0:1787698010.021782:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698010.021790:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698010.021800:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698010.021805:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698010.021821:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698010.021825:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698010.021827:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698010.021831:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5911 00000100:00080000:1.0:1787698010.021835:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698010.021838:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698010.021843:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698010.021846:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021851:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698010.021863:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:0.0:1787698010.021864:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698010.021873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021879:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698010.021883:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698010.021883:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698010.021890:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698010.021899:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698010.021900:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.021908:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787698010.021908:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698010.021914:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698010.021921:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698010.021922:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126285056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698010.021931:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126285056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698010.021944:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698010.021955:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698010.021961:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698010.021963:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:0.0:1787698010.021964:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787698010.021968:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5911 00000100:00080000:1.0:1787698010.021973:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698010.021976:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021979:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.021982:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:0.0:1787698010.021984:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698010.021985:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698010.021992:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698010.021998:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698010.022002:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:0.0:1787698010.022003:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285184:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698010.022007:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698010.022014:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698010.022019:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698010.022023:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698010.022031:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698010.022041:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698010.022044:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698010.022057:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5921) 00000800:00000100:0.0:1787698010.022059:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698010.022069:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022077:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698010.022081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.022098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.022100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698010.022102:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698010.022118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698010.022139:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022147:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698010.022159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698010.022166:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022175:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698010.022181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.022223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.022225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698010.022227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698010.022235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698010.022254:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022258:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698010.022266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698010.022270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022275:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698010.022278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.022293:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.022295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698010.022297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698010.022304:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126285184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698010.022308:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698010.022316:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126285312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698010.022320:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698010.022323:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126285440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698010.022326:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698010.022331:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022335:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698010.022342:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022346:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698010.022352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698010.022358:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698035 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022366:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698010.022372:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698010/real 1787698010] req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698010.022390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698010.022398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698010.022401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698010.022412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698010.022438:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698010.022444:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126285568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698035 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698010.022452:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698010.533489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698010.533489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698010.533497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698010.661635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698011.557491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698011.557491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698011.557492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698011.685618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698011.941524:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300585327, at 4300585344 00000100:00000040:2.0:1787698011.941613:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300585327, at 4300585344 00000100:00100000:0.0:1787698012.581508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698012.581518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698012.581522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698012.709651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698013.605564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698013.605564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698013.605569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698013.733600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698014.629579:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698014.629579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698014.629580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698014.757637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698015.141566:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698015.141579:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698015.141585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698015.141597:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5916 00000100:00080000:0.0:1787698015.141616:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698015.141618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141625:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698015.141630:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698015.141633:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141634:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141637:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698015.141639:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141658:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698015.141691:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698015.141726:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698015.141747:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698015.141751:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698015.141752:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698015.141757:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5916 00000100:00080000:0.0:1787698015.141761:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698015.141764:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141767:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141770:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698015.141773:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698015.141775:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141779:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141781:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787698015.141782:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698015.141784:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698015.141797:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126285696:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787698015.141801:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698015.141807:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698015.141816:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698015.141825:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.141829:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698015.141834:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698015.141837:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698015.141839:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698015.141839:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698015.141847:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5916 00000100:00080000:0.0:1787698015.141851:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698015.141853:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141856:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141858:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698015.141860:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698015.141863:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141864:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141866:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698015.141868:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698015.141873:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698015.141876:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698015.141883:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698015.141885:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698015.141893:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698015.141901:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698015.141904:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:1.0:1787698015.141904:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698015.141906:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698015.141911:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5916 00000100:00080000:0.0:1787698015.141913:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698015.141916:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141918:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141920:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698015.141922:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698015.141924:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141932:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141934:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698015.141937:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698015.141941:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698015.141941:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698015] req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698015.141945:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698015.141953:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698015.141958:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698015.141963:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.141969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698015.141971:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698015.141974:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698015.141976:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698015.141977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698015.141980:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5916 00000100:00080000:1.0:1787698015.141981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698015.141983:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698015.141985:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698015.141986:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141989:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.141991:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698015.141994:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698015.141994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126285696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698015.141996:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698015.141999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698015.142001:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698015.142001:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126285696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698015.142003:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698015.142008:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698015.142011:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787698015.142012:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698015.142017:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698015.142030:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698015.142041:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5926) 00000100:00100000:1.0:1787698015.142047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698015.142060:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142068:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698015.142081:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698015.142088:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142129:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698015.142135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698015] req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142146:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698015.142160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698015.142163:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698015.142167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698015.142187:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698015.142207:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142212:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698015.142220:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698015.142225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142235:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698015.142238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698015] req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142250:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698015.142255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698015.142257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698015.142259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126285952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698015.142267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698015.142279:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142284:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698015.142289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698015.142292:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142298:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698015.142301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698015] req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698015.142316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698015.142318:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698015.142320:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142324:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698015.142326:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126285824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698015.142331:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698015.142334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126285952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698015.142338:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698015.142340:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126286080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698015.142344:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698015.142349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698015.142360:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142365:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698015.142370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698015.142373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698040 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142379:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698015.142382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698015/real 1787698015] req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698015.142393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698015.142397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698015.142398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698015.142400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698015.142404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698015.142407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126286208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698040 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698015.142410:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698015.653546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698015.653550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698015.653613:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698015.781648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698016.677506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698016.677506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698016.677507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698016.805549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698017.061503:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300590447, at 4300590464 00000100:00000040:2.0:1787698017.061505:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300590447, at 4300590464 00000100:00100000:1.0:1787698017.701518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698017.701552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698017.701555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698017.829593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698018.725521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698018.725536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698018.725665:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698018.853568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698019.749513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698019.749513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698019.749513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698019.877573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698020.261533:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698020.261541:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698020.261544:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698020.261553:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5921 00000100:00080000:2.0:1787698020.261564:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698020.261567:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261571:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261574:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698020.261579:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698020.261582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261585:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261587:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698020.261590:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261601:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698020.261628:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698020.261646:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698020.261670:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698020.261673:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698020.261675:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698020.261679:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5921 00000100:00080000:2.0:1787698020.261683:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698020.261686:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261689:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261692:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698020.261694:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698020.261697:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261701:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261703:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698020.261705:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698020.261711:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698020.261713:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698020.261720:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698020.261727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126286336:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698020.261732:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698020.261739:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698020.261745:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698020.261747:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698020.261749:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698020.261752:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698020.261755:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5921 00000100:00080000:2.0:1787698020.261760:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698020.261765:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698020.261766:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261770:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261772:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698020.261774:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698020.261776:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261779:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698020.261781:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698020.261783:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698020.261787:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698020.261794:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698020.261800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261805:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.261815:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698020.261818:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698020.261820:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698020.261821:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698020.261825:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5921 00000100:00080000:2.0:1787698020.261828:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698020.261834:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261836:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261837:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698020.261838:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698020.261840:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261842:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698020.261843:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698020.261845:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698020.261849:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698020.261852:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698020.261855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698020/real 1787698020] req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698020.261859:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698020.261880:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698020.261885:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698020.261889:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698020.261889:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698020.261892:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698020.261897:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5921 00000100:00080000:2.0:1787698020.261902:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698020.261907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698020.261911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698020.261912:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698020.261913:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698020.261916:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698020.261923:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698020.261926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698020.261928:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698020.261931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:2.0:1787698020.261933:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698020.261936:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698020.261940:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698020.261947:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126286336:192.168.201.136@tcp:250:kworker.0 00000008:00100000:2.0:1787698020.261949:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698020.261955:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698020.261957:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126286336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698020.261966:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698020.261968:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698020.261980:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698020.261982:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262001:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698020.262006:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5931) 00000100:00000040:1.0:1787698020.262024:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262029:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698020.262039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698020.262044:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262051:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698020.262055:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698020/real 1787698020] req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262062:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698020.262071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698020.262073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698020.262076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698020.262083:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262088:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698020.262095:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262099:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698020.262104:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698020.262107:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262112:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698020.262117:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698020/real 1787698020] req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698020.262143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698020.262146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698020.262149:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698020.262168:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698020.262184:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262188:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698020.262195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698020.262200:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262205:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698020.262209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698020/real 1787698020] req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698020.262224:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698020.262226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698020.262228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698020.262236:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126286464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698020.262240:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698020.262244:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126286592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698020.262251:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698020.262256:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126286720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698020.262262:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698020.262270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698020.262289:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262296:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698020.262305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698020.262310:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698045 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262315:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698020.262318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698020/real 1787698020] req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698020.262329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698020.262335:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698020.262337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698020.262339:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698020.262343:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126286848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698020.262346:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126286848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698045 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698020.262350:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698020.773508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698020.773508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698020.773508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698020.901564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698021.797494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698021.797494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698021.797495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698021.925556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698022.181475:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300595567, at 4300595584 00000100:00000040:3.0:1787698022.181475:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300595567, at 4300595584 00000100:00100000:3.0:1787698022.821505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698022.821514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698022.821535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698022.949606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698023.845509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698023.845509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698023.845509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698023.973531:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698024.869485:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698024.869485:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698024.869492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698024.997565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698025.381556:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698025.381567:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698025.381571:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698025.381581:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5926 00000100:00080000:1.0:1787698025.381591:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698025.381596:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381600:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381605:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698025.381610:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698025.381614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381628:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381631:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698025.381638:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381651:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698025.381675:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698025.381690:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698025.381716:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698025.381720:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698025.381722:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698025.381726:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5926 00000100:00080000:1.0:1787698025.381730:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698025.381733:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381736:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381738:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698025.381742:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698025.381745:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381749:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381752:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698025.381755:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698025.381764:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698025.381769:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698025.381777:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698025.381782:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698025.381797:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698025.381810:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698025.381813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126286976:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698025.381815:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698025.381818:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698025.381825:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5926 00000100:00080000:1.0:1787698025.381828:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698025.381831:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381834:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381836:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698025.381839:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698025.381845:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698025.381845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381850:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381856:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698025.381863:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698025.381864:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787698025.381871:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698025.381877:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698025.381887:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698025.381897:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698025.381902:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698025.381911:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698025.381915:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698025.381917:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698025.381922:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698025.381935:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5926 00000100:00080000:1.0:1787698025.381940:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698025.381942:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381945:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381948:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698025.381952:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698025.381958:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.381962:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.381965:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698025.381968:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698025.381973:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698025.381977:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698025.381987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698025/real 1787698025] req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698025.381990:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698025.382007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698025.382009:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382022:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698025.382030:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698025.382032:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698025.382035:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698025.382038:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698025.382042:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5926 00000100:00080000:1.0:1787698025.382049:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698025.382050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698025.382055:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698025.382059:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698025.382063:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.382067:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698025.382071:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:0.0:1787698025.382078:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126286976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698025.382078:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698025.382085:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698025.382088:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787698025.382092:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126286976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698025.382094:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698025.382103:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698025.382108:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:1.0:1787698025.382109:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698025.382121:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698025.382128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698025.382138:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698025.382158:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5936) 00000100:00000040:0.0:1787698025.382161:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382167:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698025.382178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698025.382183:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382190:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698025.382194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698025/real 1787698025] req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382205:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698025.382221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698025.382225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698025.382229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698025.382242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698025.382263:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382268:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698025.382276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698025.382281:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382287:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698025.382290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698025/real 1787698025] req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698025.382306:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698025.382312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698025.382315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698025.382323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698025.382335:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382339:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698025.382345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698025.382349:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382355:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698025.382358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698025/real 1787698025] req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698025.382377:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698025.382380:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698025.382383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698025.382401:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126287104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698025.382410:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698025.382416:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126287232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698025.382445:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698025.382452:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126287360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698025.382459:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698025.382467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698025.382482:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382486:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698025.382494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698025.382499:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698050 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382505:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698025.382515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698025/real 1787698025] req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698025.382536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698025.382541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698025.382542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698025.382544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698025.382548:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698025.382552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126287488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698050 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698025.382556:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698025.893558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698025.893559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698025.893566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698026.021599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698026.917530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698026.917531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698026.917542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698027.045632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698027.301540:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300600687, at 4300600704 00000100:00000040:3.0:1787698027.301546:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300600687, at 4300600704 00000100:00100000:1.0:1787698027.941560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698027.941560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698027.941568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698028.069655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698028.965576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698028.965576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698028.965580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698029.093643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698029.989555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698029.989559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698029.989580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698030.117626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698030.501512:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698030.501520:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698030.501523:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698030.501531:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5931 00000100:00080000:2.0:1787698030.501539:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698030.501541:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501544:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501547:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698030.501551:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698030.501553:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501554:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501556:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698030.501558:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501569:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698030.501588:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698030.501598:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698030.501614:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698030.501617:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698030.501618:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698030.501621:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5931 00000100:00080000:2.0:1787698030.501623:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698030.501624:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501626:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501627:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698030.501629:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698030.501630:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501632:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501635:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698030.501638:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698030.501645:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698030.501648:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698030.501657:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698030.501663:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698030.501669:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698030.501671:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698030.501672:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698030.501677:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5931 00000100:00080000:2.0:1787698030.501681:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698030.501683:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698030.501687:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501695:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501697:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698030.501706:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698030.501716:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698030.501721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126287616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698030.501727:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501732:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698030.501735:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698030.501741:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698030.501746:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698030.501756:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698030.501760:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.501801:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501815:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698030.501821:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698030.501824:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698030.501825:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698030.501829:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5931 00000100:00080000:2.0:1787698030.501833:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698030.501846:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501850:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698030.501852:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698030.501854:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501856:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501857:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698030.501859:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698030.501863:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698030.501865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501872:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698030.501883:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698030.501889:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698030.501900:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698030.501908:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698030.501910:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698030.501911:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698030.501914:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5931 00000100:00080000:2.0:1787698030.501916:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698030.501919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501926:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501933:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698030.501941:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698030.501943:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698030.501945:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698030.501947:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698030.501949:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698030.501953:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698030.501956:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698030.501961:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698030.501967:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.501977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698030/real 1787698030] req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698030.501978:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5941) 00000100:00100000:0.0:1787698030.501998:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698030.502012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698030.502015:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698030.502018:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126287616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698030.502028:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126287616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698030.502036:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698030.502051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698030.502072:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502080:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698030.502093:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698030.502100:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502119:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698030.502124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698030/real 1787698030] req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698030.502142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698030.502144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698030.502146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698030.502154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698030.502172:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502180:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698030.502191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698030.502199:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502209:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698030.502217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698030/real 1787698030] req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698030.502234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698030.502236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698030.502238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126287872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698030.502253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698030.502269:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502274:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698030.502281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698030.502285:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502294:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698030.502299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698030/real 1787698030] req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502321:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698030.502336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698030.502339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698030.502341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698030.502351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698030.502372:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502380:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698030.502391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698030.502396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698055 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502403:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698030.502406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698030/real 1787698030] req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502415:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698030.502465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698030.502470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698030.502473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698030.502475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698030.502480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698030.502483:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126287744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698030.502487:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698030.502491:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126287872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698030.502495:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698030.502498:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126288000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698030.502502:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698030.502505:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126288128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698055 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698030.502509:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698031.013533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698031.013533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698031.013560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698031.141637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698032.037489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698032.037490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698032.037504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698032.165579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698032.421525:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300605807, at 4300605824 00000100:00000040:3.0:1787698032.421525:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300605807, at 4300605824 00000100:00100000:1.0:1787698033.061518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698033.061519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698033.061606:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698033.189559:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698034.085503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698034.085503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698034.085514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698034.213559:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698035.109507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698035.109508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698035.109511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698035.237554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698035.621491:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698035.621501:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698035.621505:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698035.621515:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5936 00000100:00080000:3.0:1787698035.621525:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698035.621528:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621533:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621536:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698035.621541:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698035.621545:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621548:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621551:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698035.621555:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621567:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698035.621592:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698035.621607:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698035.621626:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698035.621643:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698035.621646:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698035.621650:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5936 00000100:00080000:3.0:1787698035.621655:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698035.621658:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621663:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698035.621664:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698035.621666:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698035.621670:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698035.621675:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621679:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621683:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698035.621685:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698035.621690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126288256:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698035.621692:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698035.621698:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698035.621711:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698035.621718:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.621722:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698035.621726:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698035.621732:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698035.621734:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698035.621741:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5936 00000100:00000040:0.0:1787698035.621744:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698035.621745:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698035.621761:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621763:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621769:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698035.621771:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698035.621773:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621775:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621777:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698035.621778:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698035.621782:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698035.621786:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:0.0:1787698035.621793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698035.621794:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698035.621802:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698035.621808:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698035.621810:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698035.621812:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698035.621814:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698035.621816:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5936 00000100:00080000:3.0:1787698035.621821:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698035.621826:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621830:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621832:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698035.621833:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698035.621835:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621837:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621838:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698035.621840:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698035.621843:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698035.621849:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698035.621856:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698035/real 1787698035] req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698035.621858:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698035.621875:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698035.621883:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698035.621885:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698035.621886:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698035.621892:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5936 00000100:00100000:0.0:1787698035.621893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698035.621894:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698035.621899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698035.621905:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621909:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698035.621913:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698035.621916:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698035.621919:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698035.621919:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698035.621923:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698035.621927:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698035.621932:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698035.621936:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698035.621938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698035.621943:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698035.621945:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698035.621955:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698035.621956:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698035.621979:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5946) 00000100:00100000:0.0:1787698035.621983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126288256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698035.621993:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126288256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698035.622003:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698035.622026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698035.622048:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622056:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698035.622068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698035.622075:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622084:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698035.622090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698035/real 1787698035] req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698035.622117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698035.622119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698035.622121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698035.622137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698035.622169:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622176:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698035.622188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698035.622195:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622204:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698035.622210:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698035/real 1787698035] req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698035.622234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698035.622237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698035.622240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622247:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698035.622252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698035.622286:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622293:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698035.622305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698035.622311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622320:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698035.622325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698035/real 1787698035] req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622342:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698035.622348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698035.622351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698035.622355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698035.622367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622374:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698035.622384:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622400:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698035.622412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698035.622456:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698060 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622467:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698035.622473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698035/real 1787698035] req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698035.622492:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698035.622498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698035.622501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698035.622504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698035.622510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126288768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698035.622516:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126288384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698035.622523:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698035.622530:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126288512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698035.622537:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698035.622542:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126288640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698035.622554:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698035.622559:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126288768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698060 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698035.622565:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698036.133533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698036.133533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698036.133533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698036.261557:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698037.157504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698037.157504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698037.157504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698037.285588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698037.541511:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300610927, at 4300610944 00000100:00000040:3.0:1787698037.541511:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300610927, at 4300610944 00000100:00100000:0.0:1787698038.181497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698038.181497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698038.181530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698038.309591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698039.205500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698039.205500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698039.205500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698039.333562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698040.229489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698040.229489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698040.229489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698040.357559:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698040.741516:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698040.741522:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698040.741525:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698040.741533:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5941 00000100:00080000:0.0:1787698040.741541:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698040.741544:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741546:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741549:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698040.741553:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698040.741555:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741557:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741559:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698040.741561:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741571:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698040.741589:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698040.741600:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698040.741627:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698040.741630:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698040.741631:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698040.741634:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5941 00000100:00080000:0.0:1787698040.741636:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698040.741638:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741640:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741642:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698040.741644:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698040.741646:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741648:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741650:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698040.741659:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698040.741673:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.741686:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126288896:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787698040.741693:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698040.741700:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698040.741711:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698040.741716:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.741731:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741732:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698040.741743:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698040.741746:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698040.741749:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698040.741753:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5941 00000100:00080000:0.0:1787698040.741758:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698040.741760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698040.741776:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698040.741777:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741782:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698040.741787:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698040.741790:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741794:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741796:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698040.741799:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698040.741805:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698040.741808:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698040.741811:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698040/real 1787698040] req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698040.741819:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698040.741830:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.741836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698040.741838:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698040.741843:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698040.741846:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698040.741852:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5941 00000100:00080000:0.0:1787698040.741856:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698040.741857:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698040.741860:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741865:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741869:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698040.741872:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698040.741876:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741881:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741884:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698040.741889:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698040.741893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698040.741898:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698040.741899:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698040.741904:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698040.741908:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126288896:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698040.741911:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698040.741914:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126288896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698040.741921:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787698040.741922:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741931:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698040.741938:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698040.741939:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698040.741941:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698040.741944:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698040.741953:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5941 00000100:00100000:1.0:1787698040.741955:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289024:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698040.741960:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698040.741965:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698040.741971:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698040.741975:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698040.741980:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698040.741981:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698040.741985:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698040.741990:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698040.741992:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698040.741996:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698040.742000:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698040.742000:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698040.742006:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698040.742014:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698040.742014:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:0.0:1787698040.742021:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698040.742022:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698040/real 1787698040] req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698040.742030:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698040.742033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698040.742052:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698040.742078:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698040.742080:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5951) 00000100:00080000:1.0:1787698040.742081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698040.742086:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698040.742095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698040.742108:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742112:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698040.742118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698040.742121:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742137:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698040.742143:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698040/real 1787698040] req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742153:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698040.742171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698040.742174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698040.742177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742184:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698040.742187:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126289024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698040.742193:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698040.742197:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126289152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698040.742201:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698040.742206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698040.742220:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742225:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698040.742232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698040.742236:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742242:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698040.742245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698040/real 1787698040] req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698040.742264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698040.742266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698040.742268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698040.742274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742279:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698040.742284:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742288:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698040.742293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698040.742297:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698065 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742302:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698040.742304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698040/real 1787698040] req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742310:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698040.742316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698040.742320:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698040.742328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698040.742330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698040.742334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698040.742337:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126289280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698040.742341:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698040.742344:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126289408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698065 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698040.742348:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698041.253518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698041.253518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698041.253518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698041.381580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698042.277500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698042.277504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698042.277505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698042.405567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698042.661505:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300616047, at 4300616064 00000100:00000040:3.0:1787698042.661505:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300616047, at 4300616064 00000100:00100000:0.0:1787698043.301504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698043.301504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698043.301509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698043.429575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698044.325515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698044.325515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698044.325546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698044.453613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698045.349510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698045.349510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698045.349510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698045.477562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698045.861484:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698045.861492:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698045.861495:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698045.861503:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5946 00000100:00080000:0.0:1787698045.861517:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698045.861521:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861526:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861529:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698045.861534:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698045.861537:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861540:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861543:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698045.861546:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861558:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698045.861606:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698045.861616:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698045.861635:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698045.861638:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698045.861640:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698045.861644:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5946 00000100:00080000:0.0:1787698045.861662:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698045.861665:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861667:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861670:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698045.861672:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698045.861675:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861677:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861679:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787698045.861680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698045.861681:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698045.861689:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698045.861693:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698045.861698:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126289536:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698045.861702:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698045.861710:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698045.861717:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698045.861719:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698045.861721:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698045.861722:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698045.861725:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5946 00000100:00080000:0.0:1787698045.861727:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698045.861730:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698045.861730:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861732:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861734:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698045.861736:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698045.861738:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861740:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861746:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698045.861749:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698045.861753:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698045.861756:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698045.861760:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698045.861764:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698045.861772:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.861775:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698045.861779:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698045.861782:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698045.861783:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698045.861787:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5946 00000100:00080000:0.0:1787698045.861790:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698045.861792:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861795:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861797:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698045.861799:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698045.861801:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698045.861801:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698045/real 1787698045] req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698045.861804:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861807:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698045.861809:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698045.861812:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698045.861814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698045.861815:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698045.861822:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698045.861823:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698045.861832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698045.861836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698045.861840:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698045.861841:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.861848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126289536:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698045.861849:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698045.861852:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698045.861854:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698045.861855:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126289536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698045.861857:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5946 00000100:00080000:0.0:1787698045.861860:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698045.861863:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861865:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861867:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698045.861869:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698045.861871:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698045.861873:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698045.861875:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000020:00000040:1.0:1787698045.861875:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787698045.861878:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698045.861882:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698045.861885:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698045.861888:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698045.861892:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698045.861902:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.861903:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698045.861911:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5956) 00000100:00000040:1.0:1787698045.861915:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.861922:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698045.861933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698045.861941:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.861951:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698045.861955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698045/real 1787698045] req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.861962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.861968:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698045.861972:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698045.861974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698045.861976:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.861980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698045.861984:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.861989:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698045.861997:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862001:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698045.862008:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698045.862011:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862016:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698045.862019:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698045/real 1787698045] req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698045.862035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698045.862036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698045.862038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698045.862045:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698045.862055:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862059:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698045.862064:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698045.862067:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862072:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698045.862075:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698045/real 1787698045] req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862086:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698045.862089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698045.862091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698045.862093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126289920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698045.862100:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126289664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698045.862104:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698045.862107:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126289792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698045.862111:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698045.862113:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126289920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698045.862118:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698045.862122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698045.862132:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862137:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698045.862142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698045.862145:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698070 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862150:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698045.862153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698045/real 1787698045] req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698045.862163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698045.862168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698045.862169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698045.862171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698045.862175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698045.862179:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126290048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698070 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698045.862183:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698046.373525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698046.373526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698046.373526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698046.501610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698047.397487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698047.397487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698047.397487:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698047.525582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698047.781509:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300621167, at 4300621184 00000100:00000040:3.0:1787698047.781509:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300621167, at 4300621184 00000100:00100000:1.0:1787698048.421507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698048.421507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698048.421513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698048.549565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698049.445479:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698049.445489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698049.445494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698049.573562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698050.469540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698050.469540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698050.469540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698050.597552:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698050.981501:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698050.981511:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698050.981529:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698050.981539:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5951 00000100:00080000:0.0:1787698050.981548:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698050.981552:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981555:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981558:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698050.981561:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698050.981563:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981565:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981567:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698050.981569:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981578:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698050.981597:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698050.981606:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698050.981623:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698050.981626:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698050.981627:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698050.981629:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5951 00000100:00080000:0.0:1787698050.981632:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698050.981633:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981636:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698050.981638:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698050.981640:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981642:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981647:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698050.981650:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698050.981659:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698050.981663:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698050.981666:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698050.981678:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126290176:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698050.981683:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698050.981696:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.981705:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698050.981708:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698050.981717:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:1.0:1787698050.981718:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698050.981722:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698050.981729:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5951 00000100:00080000:0.0:1787698050.981744:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698050.981747:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698050.981750:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981754:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981758:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698050.981761:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698050.981765:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698050.981766:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981774:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981779:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698050.981782:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698050.981787:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698050.981792:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698050.981803:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698050.981805:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698050/real 1787698050] req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.981833:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.981867:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698050.981868:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698050.981879:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698050.981885:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698050.981886:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698050.981889:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698050.981893:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698050.981898:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698050.981898:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126290176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698050.981909:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126290176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698050.981910:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5951 00000100:00080000:0.0:1787698050.981916:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000020:00000040:1.0:1787698050.981919:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981921:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.981924:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981926:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698050.981927:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698050.981929:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698050.981933:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698050.981934:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.981938:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698050.981943:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698050.981951:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698050.981953:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290304:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787698050.981956:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698050.981967:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698050.981975:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698050.981977:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698050.981998:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698050.981999:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698050.982003:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698050.982005:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698050.982012:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5951 00000800:00000100:1.0:1787698050.982016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698050.982017:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698050.982021:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.982024:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.982026:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698050.982028:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698050.982029:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698050.982032:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698050.982036:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698050.982039:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698050.982043:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698050.982048:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698050.982049:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:0.0:1787698050.982051:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698050.982059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698050/real 1787698050] req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698050.982061:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698050.982089:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698050.982102:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5962) 00000100:00100000:1.0:1787698050.982104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698050.982116:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698050.982118:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698050.982121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698050.982130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698050.982144:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982151:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698050.982162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698050.982169:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982177:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698050.982183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698050/real 1787698050] req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698050.982206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698050.982208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698050.982210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698050.982220:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126290304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698050.982227:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698050.982233:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126290432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698050.982240:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698050.982248:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698050.982266:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982271:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698050.982279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698050.982283:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982289:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698050.982292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698050/real 1787698050] req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982303:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698050.982307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698050.982309:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698050.982311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698050.982318:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698050.982328:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982332:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698050.982337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698050.982340:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698075 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982349:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698050.982352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698050/real 1787698050] req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698050.982363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698050.982367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698050.982368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698050.982370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698050.982374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698050.982377:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126290560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698050.982381:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698050.982384:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126290688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698075 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698050.982387:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698051.493503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698051.493505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698051.493554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698051.621572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698052.517528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698052.517528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698052.517528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698052.645543:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698052.901495:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300626287, at 4300626304 00000100:00000040:2.0:1787698052.901496:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300626287, at 4300626304 00000100:00100000:2.0:1787698053.541521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698053.541522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698053.541525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698053.669561:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698054.565508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698054.565508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698054.565508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698054.693563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698055.589504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698055.589504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698055.589506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698055.717587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698056.101528:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698056.101535:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698056.101538:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698056.101546:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5957 00000100:00080000:3.0:1787698056.101555:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698056.101557:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101562:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698056.101566:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698056.101568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101570:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101586:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698056.101588:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101596:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698056.101616:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698056.101626:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698056.101642:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698056.101644:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698056.101646:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698056.101648:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5957 00000100:00080000:3.0:1787698056.101651:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698056.101653:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101654:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101656:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698056.101658:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698056.101660:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101662:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101664:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698056.101666:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698056.101673:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698056.101676:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698056.101680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698056.101685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698056.101699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126290816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698056.101708:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698056.101723:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698056.101724:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698056.101730:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698056.101733:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698056.101740:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5957 00000100:00000040:1.0:1787698056.101741:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698056.101745:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698056.101748:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101751:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101752:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698056.101754:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698056.101756:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101758:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101759:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698056.101761:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698056.101764:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698056.101768:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698056.101775:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698056.101781:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698056.101787:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698056.101789:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698056.101791:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698056.101793:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5957 00000100:00080000:3.0:1787698056.101796:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698056.101803:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698056.101804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101813:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698056.101817:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698056.101821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698056.101825:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101830:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101833:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698056.101836:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698056.101840:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698056.101850:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698056.101850:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698056.101857:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698056.101866:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698056.101867:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787698056.101875:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698056.101880:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698056.101893:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698056.101894:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291072:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698056.101901:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698056.101904:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698056.101909:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698056.101910:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5957 00000100:00080000:3.0:1787698056.101918:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698056.101920:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101925:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698056.101930:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787698056.101931:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000020:00000040:3.0:1787698056.101932:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698056.101937:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698056.101939:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698056.101942:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698056.101943:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698056.101950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291200:192.168.201.136@tcp:8:kworker.0 00000100:00080000:3.0:1787698056.101954:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698056.101960:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698056.101962:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698056.101965:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698056.101970:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698056.101977:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698056.101979:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787698056.101991:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698056.102013:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5967) 00000100:00000040:1.0:1787698056.102040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698056.102049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698056.102062:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698056.102070:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698056.102081:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698056.613529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698056.613532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698056.613554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698056.741566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698057.637542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698057.637542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698057.637567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698057.765594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698058.021521:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300631407, at 4300631424 00000100:00000040:3.0:1787698058.021521:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300631407, at 4300631424 00000100:00100000:2.0:1787698058.661564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698058.661567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698058.661574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698058.789637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787698059.173776:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787698059.173835:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787698059.173872:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698059.173921:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698059.173942:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698059.173952:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698059.173963:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698059.173971:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698059.173981:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698059.173989:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698059.173998:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698059.174006:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698081 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174112:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698059] req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698059.174149:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.174171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.174177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698059.174182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174191:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291328:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698059.174199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698059] req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698059.174218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.174225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.174228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698059.174232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291200:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698059.174245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698059] req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698059.174270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.174277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.174281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698059.174285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126290944:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698059.174299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698059] req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698059.174317:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.174325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.174328:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698059.174332:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291072:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698059.174345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698056/real 1787698059] req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698059.174400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698059.174407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698059.174410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698059.174413:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698059.174457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126290816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698059.174464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126291328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698059.174478:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698059.174488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126291200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698059.174495:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698059.174500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126290944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698059.174507:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698059.174511:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126291072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698059.174518:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698059.174522:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126290816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698081 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698059.174530:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698059.685525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698059.685526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698059.685555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698059.813602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698060.709554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698060.709554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698060.709590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698060.837638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698061.221575:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698061.221586:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698061.221590:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698061.221604:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5962 00000100:00080000:1.0:1787698061.221616:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698061.221619:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.221624:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221628:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698061.221633:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698061.221636:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.221639:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221642:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698061.221646:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221659:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698061.221688:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698061.221705:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698061.221731:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698061.221735:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698061.221737:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698061.221741:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5962 00000100:00080000:1.0:1787698061.221745:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698061.221748:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.221750:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221752:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698061.221755:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698061.221758:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.221761:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221762:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698061.221765:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698061.221775:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698061.221776:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698061.221782:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698061.221797:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698061.221800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126291456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698061.221815:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698061.221824:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698061.221827:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698061.221829:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698061.221833:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5962 00000100:00080000:1.0:1787698061.221836:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698061.221838:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698061.221844:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698061.221844:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221848:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698061.221852:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698061.221859:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698061.221862:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698061.221866:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221869:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698061.221872:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698061.221877:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698061.221882:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698061.221891:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698061.221901:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698061.221910:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698061.221912:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698061.221916:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698061.221918:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698061.221925:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5962 00000100:00080000:1.0:1787698061.221929:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698061.221933:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698061.221934:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.221940:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221945:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698061.221949:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698061.221951:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.221960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.221963:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698061.221965:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698061.221969:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698061.221976:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698061.221987:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698061.222003:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698061.222011:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698061.222014:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698061.222016:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698061.222020:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5962 00000100:00080000:1.0:1787698061.222024:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698061.222026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698061/real 1787698061] req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698061.222030:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.222036:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.222047:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698061.222049:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:0.0:1787698061.222053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698061.222056:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698061.222063:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698061.222068:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787698061.222073:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698061.222074:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698061.222083:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698061.222087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787698061.222089:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698061.222094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:1.0:1787698061.222104:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698061.222105:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698061.222120:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126291456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698061.222137:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5972) 00000100:00000040:0.0:1787698061.222139:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126291456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698061.222156:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698061.222174:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698061.222197:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222205:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698061.222216:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698061.222223:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222244:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698061.222250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698061/real 1787698061] req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698061.222277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698061.222281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698061.222290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698061.222304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222325:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698061.222337:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222344:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698061.222355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698061.222361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222371:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698061.222381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698061/real 1787698061] req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698061.222410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698061.222413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698061.222439:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698061.222453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698061.222480:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222488:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698061.222498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698061.222508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222518:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698061.222523:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698061/real 1787698061] req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222553:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698061.222561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698061.222564:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698061.222569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698061.222581:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698061.222599:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222608:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698061.222617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698061.222623:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698086 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222632:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698061.222637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698061/real 1787698061] req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698061.222672:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698061.222681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698061.222685:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698061.222689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698061.222697:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126291968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698061.222702:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126291584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698061.222711:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698061.222717:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126291712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698061.222724:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698061.222729:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126291840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698061.222740:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698061.222745:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126291968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698086 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698061.222752:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698061.733553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698061.733553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698061.733665:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698061.861690:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698062.757556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698062.757575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698062.760362:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698062.885816:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698063.141548:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300636527, at 4300636544 00000100:00000040:3.0:1787698063.143826:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300636527, at 4300636546 00000100:00100000:3.0:1787698063.781485:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698063.781486:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698063.781552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698063.909613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698064.805551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698064.805551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698064.805552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698064.933563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698065.829592:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698065.829941:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698065.830024:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698065.957608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698066.341533:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698066.341545:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698066.341550:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698066.341561:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5967 00000100:00080000:1.0:1787698066.341573:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698066.341577:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341582:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341587:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698066.341591:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698066.341595:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341597:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341600:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698066.341603:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341618:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698066.341646:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698066.341661:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698066.341683:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698066.341687:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698066.341689:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698066.341693:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5967 00000100:00080000:1.0:1787698066.341697:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698066.341701:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341705:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341707:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698066.341710:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698066.341712:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698066.341716:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341723:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341728:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698066.341730:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698066.341733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126292096:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787698066.341741:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698066.341748:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698066.341761:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698066.341762:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.341787:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341794:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698066.341804:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698066.341808:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698066.341810:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698066.341814:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5967 00000100:00080000:1.0:1787698066.341818:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698066.341820:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341824:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341828:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341831:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698066.341834:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698066.341838:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698066.341840:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341847:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341850:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698066.341852:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698066.341857:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698066.341863:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698066.341873:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698066.341883:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.341888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698066/real 1787698066] req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698066.341901:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698066.341906:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698066.341908:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698066.341912:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698066.341914:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5967 00000100:00080000:1.0:1787698066.341921:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698066.341925:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698066.341927:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.341934:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341937:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00080000:0.0:1787698066.341938:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787698066.341941:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:0.0:1787698066.341944:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698066.341948:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698066.341950:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698066.341954:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.341959:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:0.0:1787698066.341963:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126292096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698066.341966:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698066.341973:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698066.341979:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698066.341980:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126292096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698066.341990:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787698066.341993:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698066.342002:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698066.342021:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698066.342024:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698066.342028:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698066.342029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698066.342036:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5967 00000100:00080000:1.0:1787698066.342043:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698066.342049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698066.342050:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.342057:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.342060:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698066.342063:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698066.342066:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698066.342068:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698066.342072:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787698066.342073:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698066.342079:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698066.342088:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787698066.342091:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698066.342096:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698066.342108:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698066.342112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698066.342121:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342125:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698066.342133:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5977) 00000100:00000040:0.0:1787698066.342141:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698066.342149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698066/real 1787698066] req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342159:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698066.342169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698066.342170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698066.342173:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698066.342181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698066.342237:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342246:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698066.342258:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698066.342263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342270:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698066.342273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698066/real 1787698066] req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698066.342290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698066.342291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698066.342294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698066.342301:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698066.342320:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342324:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698066.342331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698066.342335:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342340:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698066.342343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698066/real 1787698066] req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698066.342359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698066.342360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698066.342362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698066.342369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126292224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698066.342375:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698066.342379:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126292352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698066.342390:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698066.342393:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126292480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698066.342397:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698066.342402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698066.342413:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342417:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698066.342441:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698066.342446:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698091 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342451:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698066.342455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698066/real 1787698066] req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698066.342467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698066.342471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698066.342473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698066.342475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698066.342483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698066.342487:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126292608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698091 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698066.342491:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698066.853581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698066.853584:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698066.853627:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698066.981716:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698067.877498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698067.877549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698067.877557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698068.005659:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698068.261557:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300641647, at 4300641664 00000100:00000040:3.0:1787698068.261568:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300641647, at 4300641664 00000100:00100000:1.0:1787698068.901522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698068.901522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698068.901528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698069.029620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698069.925553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698069.925681:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698069.926580:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698070.053913:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698070.949487:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698070.949515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698070.949537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698071.077622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698071.461630:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698071.461640:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698071.461645:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698071.461655:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5972 00000100:00080000:2.0:1787698071.461666:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698071.461669:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.461673:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.461677:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698071.461681:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698071.461684:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.461687:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.461690:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698071.461693:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698071.461706:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698071.461732:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698071.461805:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.461825:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698071.461829:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698071.461831:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698071.461835:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5972 00000100:00080000:2.0:1787698071.461839:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698071.461842:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.461844:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.461846:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698071.461849:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698071.461851:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.461854:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.461857:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698071.461859:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698071.461867:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698071.461872:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698071.461882:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698071.461892:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.461901:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698071.461903:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698071.461905:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698071.461907:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.461909:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5972 00000100:00080000:2.0:1787698071.461913:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698071.461916:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.461918:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698071.461921:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126292736:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787698071.461921:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698071.461924:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698071.461926:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.461929:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.461931:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:0.0:1787698071.461949:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.461958:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698071.462005:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698071.462022:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462054:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698071/real 1787698071] req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698071.462114:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698071.462119:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698071.462124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462132:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126292736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698071.462166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462175:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292864:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698071.462179:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698071.462184:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698071.462188:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698071.462189:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462196:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698071.462198:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698071.462208:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698071.462212:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462215:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.462221:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698071.462224:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:0.0:1787698071.462225:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:2.0:1787698071.462226:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698071.462229:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5972 00000100:00100000:0.0:1787698071.462231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698071/real 1787698071] req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.462233:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698071.462235:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.462238:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.462240:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:0.0:1787698071.462242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698071.462242:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698071.462245:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.462248:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.462250:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:0.0:1787698071.462251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.462253:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698071.462257:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698071.462261:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698071.462268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698071.462269:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698071.462276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698071.462278:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.462285:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698071.462289:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292864:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698071.462289:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698071.462290:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698071.462294:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5972 00000100:00000040:0.0:1787698071.462295:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126292736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698071.462297:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698071.462299:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.462302:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.462303:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698071.462304:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:2.0:1787698071.462305:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698071.462308:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698071.462310:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698071.462312:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698071.462314:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698071.462319:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126292864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698071.462319:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698071.462322:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:0.0:1787698071.462327:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698071.462329:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698071.462336:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698071.462347:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5982) 00000100:00000040:0.0:1787698071.462369:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462377:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698071.462391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698071.462399:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462416:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698071.462439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698071/real 1787698071] req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462451:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698071.462481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698071.462484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698071.462488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126292992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698071.462501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698071.462524:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462531:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698071.462546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698071.462552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462562:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698071.462568:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698071/real 1787698071] req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462598:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698071.462612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698071.462615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698071.462620:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698071.462633:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126292992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698071.462642:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698071.462648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126293120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698071.462654:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698071.462663:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698071.462685:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462692:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698071.462704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698071.462711:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698096 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462721:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698071.462725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698071/real 1787698071] req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698071.462759:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698071.462765:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698071.462768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698071.462772:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698071.462779:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698071.462785:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126293248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698096 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698071.462793:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698071.973568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698071.973662:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698071.973732:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698072.101590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698072.997503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698072.997526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698072.998486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698073.125703:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698073.381538:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300646767, at 4300646784 00000100:00000040:3.0:1787698073.381548:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300646767, at 4300646784 00000100:00100000:0.0:1787698074.021556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698074.021603:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698074.021656:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698074.149592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698075.045528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698075.045546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698075.045628:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698075.173604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698076.069528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698076.069528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698076.069529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698076.197607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698076.581519:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698076.581526:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698076.581530:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698076.581537:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5977 00000100:00080000:1.0:1787698076.581546:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698076.581548:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581552:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581554:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698076.581558:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698076.581560:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581562:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581564:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698076.581568:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581579:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698076.581605:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698076.581620:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698076.581650:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698076.581653:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698076.581654:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698076.581657:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5977 00000100:00080000:1.0:1787698076.581660:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698076.581662:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581665:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581667:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698076.581669:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698076.581671:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581673:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581675:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698076.581677:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698076.581683:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698076.581685:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698076.581691:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698076.581703:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698076.581706:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126293376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698076.581716:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698076.581721:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698076.581723:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698076.581724:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698076.581727:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5977 00000100:00080000:1.0:1787698076.581729:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698076.581731:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581733:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581735:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698076.581738:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698076.581740:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698076.581742:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581747:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581750:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698076.581754:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698076.581758:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698076.581760:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698076.581766:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698076.581774:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581775:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698076.581780:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698076.581783:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698076.581785:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698076.581791:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5977 00000100:00080000:1.0:1787698076.581795:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698076.581798:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581801:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581803:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698076.581805:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698076.581808:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581813:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581815:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698076.581818:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698076.581823:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698076.581825:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581829:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698076.581841:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698076.581845:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698076.581862:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698076.581872:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698076.581874:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698076.581875:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698076.581878:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5977 00000100:00080000:1.0:1787698076.581882:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698076.581884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698076.581888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698076/real 1787698076] req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698076.581889:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581893:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698076.581896:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698076.581900:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698076.581905:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698076.581908:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698076.581911:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698076.581915:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698076.581918:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698076.581921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698076.581927:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698076.581940:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698076.581941:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698076.581965:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5987) 00000100:00080000:0.0:1787698076.581967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698076.581972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698076.581978:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.581986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126293376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698076.581992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126293376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698076.582002:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698076.582016:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582025:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698076.582038:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582045:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698076.582074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698076.582086:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582097:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698076.582103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698076/real 1787698076] req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582123:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698076.582130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698076.582133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698076.582137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698076.582150:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698076.582169:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582177:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698076.582186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698076.582192:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582211:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698076.582217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698076/real 1787698076] req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698076.582244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698076.582247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698076.582251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698076.582264:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698076.582281:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582287:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698076.582298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698076.582304:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582314:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698076.582323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698076/real 1787698076] req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698076.582350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698076.582353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698076.582357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698076.582370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698076.582388:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582395:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698076.582404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698076.582409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698101 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582434:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698076.582439:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698076/real 1787698076] req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698076.582470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698076.582477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698076.582480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698076.582484:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698076.582491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126293888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698076.582497:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126293504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698076.582505:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698076.582510:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126293632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698076.582517:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698076.582522:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126293760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698076.582528:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698076.582533:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126293888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698101 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698076.582539:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698077.093506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698077.093507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698077.093516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698077.221624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698078.117554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698078.117567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698078.117568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698078.245603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698078.501485:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300651887, at 4300651904 00000100:00000040:3.0:1787698078.501515:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300651887, at 4300651904 00000100:00100000:1.0:1787698079.141526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698079.141532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698079.141566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698079.269646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698080.165573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698080.165577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698080.165618:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698080.293600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698081.189503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698081.189565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698081.189574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698081.317683:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698081.701555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698081.701566:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698081.701570:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698081.701582:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5982 00000100:00080000:3.0:1787698081.701600:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698081.701603:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701606:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701630:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698081.701635:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698081.701637:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701639:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701641:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698081.701644:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701656:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698081.701683:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698081.701695:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698081.701712:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698081.701714:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698081.701716:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698081.701719:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5982 00000100:00080000:3.0:1787698081.701722:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698081.701724:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701728:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698081.701731:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698081.701733:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701735:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701737:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698081.701738:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698081.701746:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698081.701747:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698081.701754:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698081.701762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126294016:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698081.701772:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698081.701784:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698081.701794:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787698081.701799:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698081.701800:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698081.701810:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698081.701814:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698081.701817:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5982 00000100:00080000:3.0:1787698081.701822:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698081.701824:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701826:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701828:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698081.701830:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698081.701832:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701834:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701835:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698081.701838:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698081.701841:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698081.701845:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698081.701853:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698081.701859:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698081.701866:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787698081.701867:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698081.701871:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698081.701878:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698081.701883:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5982 00000100:00080000:3.0:1787698081.701887:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698081.701890:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698081.701891:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701898:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701902:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698081.701905:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698081.701908:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.701910:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.701912:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698081.701913:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698081.701917:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698081.701920:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698081.701927:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698081.701933:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.701938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698081/real 1787698081] req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698081.701953:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698081.701955:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698081.701956:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698081.701990:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5982 00000100:00100000:0.0:1787698081.701992:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698081.701996:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698081.702003:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.702008:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698081.702012:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698081.702012:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698081.702017:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:0.0:1787698081.702026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698081.702026:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698081.702029:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698081.702032:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698081.702036:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698081.702037:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698081.702045:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787698081.702048:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698081.702049:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698081.702060:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698081.702064:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126294016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698081.702071:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702079:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126294016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698081.702081:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5992) 00000020:00000040:0.0:1787698081.702100:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698081.702119:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698081.702144:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702152:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698081.702165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698081.702172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702183:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698081.702188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698081/real 1787698081] req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698081.702206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698081.702208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698081.702210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702215:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698081.702224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702230:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698081.702240:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702255:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698081.702264:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698081.702270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702280:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698081.702285:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698081/real 1787698081] req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698081.702314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698081.702317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698081.702321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698081.702334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702349:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698081.702363:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702368:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698081.702378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698081.702384:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702391:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698081.702395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698081/real 1787698081] req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702407:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698081.702416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698081.702438:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698081.702440:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698081.702448:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698081.702469:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702474:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698081.702482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698081.702486:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698106 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702492:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698081.702496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698081/real 1787698081] req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698081.702508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698081.702515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698081.702517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698081.702520:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698081.702524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698081.702528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126294144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698081.702533:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698081.702541:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126294272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698081.702555:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698081.702559:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126294400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698081.702563:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698081.702566:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126294528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698106 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698081.702571:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698082.213525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698082.213525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698082.213525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698082.341589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698083.237525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698083.237525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698083.237525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698083.365607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698083.621519:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300657007, at 4300657024 00000100:00000040:2.0:1787698083.621520:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300657007, at 4300657024 00000100:00100000:2.0:1787698084.261534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698084.261537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698084.261548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698084.389562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698085.285518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698085.285518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698085.285628:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698085.413578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698086.309536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698086.309536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698086.309536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698086.437615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698086.821523:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698086.821533:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698086.821538:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698086.821548:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5987 00000100:00080000:3.0:1787698086.821558:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698086.821562:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821566:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821569:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698086.821574:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698086.821578:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821581:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821584:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698086.821588:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821601:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698086.821625:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698086.821653:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698086.821676:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698086.821679:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698086.821681:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698086.821685:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5987 00000100:00080000:3.0:1787698086.821689:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698086.821691:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821694:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821696:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698086.821699:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698086.821702:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821705:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821707:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698086.821709:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698086.821718:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698086.821724:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698086.821727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698086.821736:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698086.821748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126294656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698086.821752:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698086.821763:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698086.821766:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698086.821768:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698086.821779:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5987 00000100:00000040:0.0:1787698086.821780:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698086.821785:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698086.821793:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821803:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821806:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821809:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698086.821812:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698086.821815:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821817:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821820:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698086.821822:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698086.821827:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787698086.821833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821834:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698086.821847:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698086.821852:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698086.821865:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698086.821895:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698086.821898:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698086.821899:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698086.821903:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5987 00000100:00080000:3.0:1787698086.821907:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698086.821910:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.821913:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821916:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698086.821918:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698086.821923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698086.821924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698086/real 1787698086] req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698086.821936:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.821941:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698086.821944:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698086.821950:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698086.821951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698086.821957:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698086.821970:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698086.821989:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.821992:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698086.822005:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698086.822009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698086.822009:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698086.822013:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698086.822016:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698086.822021:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5987 00000100:00000040:0.0:1787698086.822026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698086.822028:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698086.822034:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.822041:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698086.822044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126294656:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698086.822045:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698086.822049:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698086.822057:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126294656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698086.822057:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698086.822063:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698086.822068:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698086.822074:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698086.822078:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:3.0:1787698086.822080:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698086.822086:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698086.822095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698086.822097:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698086.822109:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698086.822114:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698086.822123:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (5997) 00000100:00000040:0.0:1787698086.822130:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822139:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698086.822153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698086.822161:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822171:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698086.822176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698086/real 1787698086] req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698086.822216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698086.822220:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698086.822224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698086.822236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698086.822256:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822263:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698086.822276:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698086.822282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822292:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698086.822297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698086/real 1787698086] req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698086.822334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698086.822337:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698086.822340:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822347:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126294912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698086.822352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698086.822373:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822380:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698086.822392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698086.822398:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822407:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698086.822413:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698086/real 1787698086] req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698086.822477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698086.822480:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698086.822483:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698086.822495:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126294784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698086.822503:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698086.822509:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126294912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698086.822516:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698086.822521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126295040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698086.822528:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698086.822535:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698086.822563:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822571:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698086.822582:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698086.822589:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698111 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822598:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698086.822603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698086/real 1787698086] req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698086.822622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698086.822629:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698086.822632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698086.822636:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698086.822643:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698086.822648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126295168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698111 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698086.822655:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698087.333527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698087.333527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698087.333732:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698087.461585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698088.357522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698088.357523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698088.357545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698088.485570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698088.741489:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300662127, at 4300662144 00000100:00000040:2.0:1787698088.741497:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300662127, at 4300662144 00000100:00100000:1.0:1787698089.381520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698089.381521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698089.381521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698089.509606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698090.405498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698090.405500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698090.405501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698090.533605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698091.429564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698091.429564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698091.429686:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698091.557568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698091.941471:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698091.941478:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698091.941481:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698091.941488:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5992 00000100:00080000:3.0:1787698091.941496:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698091.941498:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941501:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941504:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698091.941508:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698091.941510:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941511:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941514:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698091.941516:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941527:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698091.941547:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698091.941559:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698091.941578:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698091.941582:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698091.941583:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698091.941603:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5992 00000100:00000040:0.0:1787698091.941604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698091.941608:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698091.941614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941619:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698091.941621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126295296:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698091.941623:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698091.941628:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698091.941633:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941637:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698091.941639:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698091.941646:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698091.941649:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698091.941655:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941658:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698091.941671:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698091.941680:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698091.941688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698091.941696:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698091.941700:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698091.941702:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698091.941706:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698091.941708:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5992 00000100:00080000:3.0:1787698091.941715:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698091.941721:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941727:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698091.941729:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698091.941732:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941745:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698091.941748:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698091/real 1787698091] req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698091.941749:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698091.941754:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698091.941768:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698091.941770:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698091.941777:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698091.941781:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698091.941802:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698091.941805:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698091.941812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698091.941820:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698091.941824:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.941831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126295296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698091.941843:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126295296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698091.941845:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698091.941851:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698091.941854:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698091.941858:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5992 00000100:00080000:3.0:1787698091.941861:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698091.941863:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941866:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941868:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698091.941871:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698091.941877:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941881:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941883:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000020:00000040:0.0:1787698091.941884:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698091.941885:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698091.941892:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698091.941896:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698091.941902:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698091.941905:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698091.941912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698091.941927:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.941934:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698091.941935:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698091.941941:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698091.941943:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698091.941949:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5992 00000100:00080000:3.0:1787698091.941953:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698091.941958:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941966:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698091.941967:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698091.941970:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698091.941971:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698091.941973:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000800:00000100:0.0:1787698091.941978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698091.941978:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698091.941985:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698091.941988:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698091.941990:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698091.941997:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698091.942001:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787698091.942006:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698091/real 1787698091] req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698091.942016:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6003) 00000100:00100000:0.0:1787698091.942024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942031:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698091.942036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698091.942038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698091.942040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698091.942054:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698091.942068:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942072:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698091.942079:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698091.942083:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942089:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698091.942092:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698091/real 1787698091] req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698091.942107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698091.942109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698091.942111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698091.942118:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126295424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698091.942126:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698091.942130:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126295552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698091.942134:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698091.942139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698091.942152:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942157:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698091.942166:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698091.942172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942181:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698091.942186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698091/real 1787698091] req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942205:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698091.942211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698091.942212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698091.942215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698091.942227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942233:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698091.942240:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942245:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698091.942252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698091.942256:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698116 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942262:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698091.942265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698091/real 1787698091] req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698091.942277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698091.942281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698091.942282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698091.942284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698091.942288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126295808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698091.942296:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126295680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698091.942301:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698091.942305:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126295808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698116 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698091.942308:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698092.453528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698092.453532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698092.454439:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698092.581555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698093.477514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698093.477514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698093.477515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698093.605562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698093.861501:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300667247, at 4300667264 00000100:00000040:2.0:1787698093.861590:0:88373:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300667247, at 4300667264 00000100:00100000:1.0:1787698094.501511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698094.501514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698094.501539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698094.629616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698095.525514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698095.525514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698095.525517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698095.653582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698096.549503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698096.549506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698096.549593:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698096.677572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698097.061518:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698097.061528:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698097.061532:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698097.061542:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 5998 00000100:00080000:3.0:1787698097.061552:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698097.061555:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061563:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698097.061567:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698097.061571:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061573:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061577:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698097.061580:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061593:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698097.061617:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698097.061631:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698097.061652:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698097.061655:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698097.061657:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698097.061662:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5998 00000100:00080000:3.0:1787698097.061666:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698097.061669:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698097.061681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698097.061690:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061695:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698097.061713:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698097.061717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061719:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061722:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00100000:0.0:1787698097.061727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126295936:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698097.061727:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698097.061739:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698097.061745:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698097.061758:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698097.061760:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698097.061778:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061784:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698097.061795:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698097.061798:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698097.061801:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698097.061805:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5998 00000800:00000100:0.0:1787698097.061808:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698097.061810:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698097.061816:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061820:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061822:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:0.0:1787698097.061825:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698097.061826:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698097.061835:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061839:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061841:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698097.061844:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698097.061854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698097.061856:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698097.061864:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698097.061870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698097.061878:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698097.061881:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698097.061893:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.061897:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698097.061903:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698097.061909:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698097.061910:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698097.061918:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698097.061923:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126295936:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698097.061923:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698097.061932:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5998 00000100:00000040:0.0:1787698097.061934:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126295936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698097.061939:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000020:00000040:0.0:1787698097.061943:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061945:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061952:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698097.061955:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698097.061958:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698097.061961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.061968:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.061973:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:0.0:1787698097.061975:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296064:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698097.061980:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698097.061986:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787698097.061991:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698097.061992:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698097.062003:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698097.062006:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698097.062018:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698097.062022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698097.062038:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698097.062048:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698097.062052:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:0.0:1787698097.062053:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787698097.062054:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698097.062062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698097.062063:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 5998 00000100:00080000:3.0:1787698097.062070:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698097.062076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698097.062077:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698097.062085:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.062088:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698097.062091:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698097.062098:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698097.062105:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698097.062105:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698097.062111:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698097.062116:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:0.0:1787698097.062121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698097.062123:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698097.062125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698097.062131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698097.062133:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698097.062141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698097.062151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698097.062152:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698097.062164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698097.062174:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062179:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062194:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698097.062196:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6008) 00000800:00000100:0.0:1787698097.062204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698097.062210:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062217:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698097.062220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062226:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.062240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698097.062242:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698097.062245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698097.062252:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126296064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698097.062257:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698097.062261:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126296192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698097.062268:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698097.062275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698097.062296:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062304:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698097.062314:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698097.062319:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062326:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698097.062329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.062354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698097.062355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698097.062358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698097.062370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698097.062389:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062396:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698097.062407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698097.062413:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698122 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062437:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698097.062441:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698097/real 1787698097] req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698097.062458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698097.062463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698097.062465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698097.062467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698097.062471:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698097.062474:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126296320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698097.062479:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698097.062483:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126296448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698122 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698097.062487:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698097.573524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698097.573525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698097.573526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698097.701630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698098.597559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698098.597566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698098.597570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698098.725614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698098.981566:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300672367, at 4300672384 00000100:00000040:3.0:1787698098.981566:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300672367, at 4300672384 00000100:00100000:1.0:1787698099.621582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698099.621589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698099.621592:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698099.749578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698100.645514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698100.645514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698100.645561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698100.773591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698101.669549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698101.669549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698101.669551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698101.797611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698102.181544:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698102.181553:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698102.181557:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698102.181568:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6003 00000100:00080000:3.0:1787698102.181578:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698102.181582:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698102.181586:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.181589:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698102.181594:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698102.181597:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698102.181600:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.181603:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698102.181606:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698102.181636:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698102.181660:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698102.181675:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698102.181705:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698102.181709:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698102.181711:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698102.181717:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6003 00000100:00080000:3.0:1787698102.181722:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698102.181727:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698102.181730:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.181748:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:0.0:1787698102.181754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698102.181754:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698102.181771:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698102.181778:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698102.181782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126296576:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698102.181784:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698102.181793:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698102.181803:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698102.181811:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698102.181820:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698102.181829:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698102.181836:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698102.181852:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698102.181863:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698102.181867:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698102.181870:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698102.181874:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6003 00000100:00080000:3.0:1787698102.181879:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698102.181883:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:0.0:1787698102.181890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698102.181891:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.181895:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698102.181897:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698102.181911:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698102.181915:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.181919:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698102.181922:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698102.181928:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698102.181930:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698102.181944:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698102.181959:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698102.181970:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698102/real 1787698102] req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698102.181977:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698102.182002:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698102.182010:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698102.182010:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698102.182015:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698102.182028:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6003 00000100:00000040:0.0:1787698102.182035:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698102.182036:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698102.182050:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698102.182061:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698102.182063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787698102.182066:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698102.182072:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:0.0:1787698102.182076:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698102.182083:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698102.182087:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698102.182092:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.182099:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698102.182109:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698102.182117:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126296576:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698102.182118:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698102.182132:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698102.182137:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126296576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698102.182148:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787698102.182154:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698102.182163:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698102.182173:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698102.182184:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:0.0:1787698102.182186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698102.182188:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698102.182204:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6003 00000100:00100000:0.0:1787698102.182208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296704:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698102.182212:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698102.182220:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698102.182226:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698102.182228:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.182240:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:0.0:1787698102.182245:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698102.182248:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698102.182256:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:0.0:1787698102.182262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698102.182271:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698102.182272:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698102.182279:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698102.182287:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698102.182299:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698102.182307:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698102.182308:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698102.182321:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698102/real 1787698102] req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698102.182323:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698102.182369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698102.182374:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698102.182443:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698102.182446:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698102.182449:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698102.182469:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6013) 00000100:00000040:0.0:1787698102.182476:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182487:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698102.182506:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182514:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698102.182524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698102.182529:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182537:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698102.182549:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698102/real 1787698102] req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698102.182578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698102.182581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698102.182585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182593:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698102.182599:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182608:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698102.182635:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182643:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698102.182656:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698102.182663:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182674:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698102.182679:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698102/real 1787698102] req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182702:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182710:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698102.182718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698102.182721:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698102.182725:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182733:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126296960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698102.182739:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126296704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698102.182748:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698102.182754:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126296832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698102.182762:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698102.182767:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126296960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698102.182773:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698102.182782:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182790:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698102.182811:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182818:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698102.182830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698102.182836:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698127 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182846:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698102.182852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698102/real 1787698102] req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698102.182871:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698102.182878:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698102.182882:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698102.182885:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698102.182893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698102.182898:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126297088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698127 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698102.182906:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698102.693475:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698102.693510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698102.693645:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698102.821574:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698103.717510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698103.717516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698103.717532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698103.845599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698104.101587:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300677487, at 4300677504 00000100:00000040:3.0:1787698104.101685:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300677487, at 4300677504 00000100:00100000:2.0:1787698104.741529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698104.741529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698104.741579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698104.869639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698105.765518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698105.765518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698105.765540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698105.893527:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698106.789524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698106.789525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698106.789525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698106.917529:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698107.301495:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698107.301505:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698107.301536:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698107.301544:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6008 00000100:00080000:3.0:1787698107.301553:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698107.301555:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301558:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301561:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698107.301565:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698107.301567:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301569:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301571:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698107.301573:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301581:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698107.301600:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698107.301610:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698107.301627:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698107.301629:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698107.301630:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698107.301633:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6008 00000100:00080000:3.0:1787698107.301635:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698107.301636:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301639:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698107.301641:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698107.301643:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301644:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301649:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698107.301652:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698107.301659:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698107.301662:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698107.301669:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698107.301678:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698107.301679:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698107.301694:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698107.301696:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698107.301697:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698107.301700:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6008 00000100:00080000:3.0:1787698107.301704:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698107.301706:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301710:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301713:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698107.301715:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698107.301717:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698107.301721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126297216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698107.301721:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301725:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698107.301730:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698107.301733:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698107.301736:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698107.301742:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698107.301754:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698107.301771:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.301775:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698107.301784:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698107.301788:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698107.301790:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698107.301795:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6008 00000100:00080000:3.0:1787698107.301798:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698107.301800:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301803:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301804:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698107.301805:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698107.301807:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301809:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301810:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000800:00000100:0.0:1787698107.301814:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698107.301815:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698107.301821:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698107.301825:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698107.301835:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698107.301835:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698107.301865:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698107.301870:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698107.301872:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698107.301873:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698107.301880:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6008 00000100:00080000:3.0:1787698107.301883:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698107.301885:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301887:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301890:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:0.0:1787698107.301891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698107/real 1787698107] req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698107.301892:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698107.301897:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698107.301901:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698107.301905:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698107.301908:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698107.301914:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698107.301917:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698107.301922:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698107.301924:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698107.301950:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.301954:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698107.301959:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6018) 00000100:00080000:0.0:1787698107.301967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698107.301972:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698107.301977:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.301984:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126297216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698107.301990:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126297216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698107.301999:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698107.302020:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698107.302042:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302049:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698107.302060:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698107.302066:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302075:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698107.302081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698107/real 1787698107] req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698107.302105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698107.302108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698107.302111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698107.302123:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698107.302150:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302157:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698107.302167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698107.302173:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302181:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698107.302186:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698107/real 1787698107] req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698107.302213:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698107.302215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698107.302219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698107.302231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302239:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698107.302257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302264:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698107.302273:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698107.302279:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302287:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698107.302291:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698107/real 1787698107] req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698107.302316:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698107.302319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698107.302322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698107.302334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302341:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698107.302350:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302374:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698107.302386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698107.302393:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698132 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302401:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698107.302406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698107/real 1787698107] req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698107.302450:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698107.302456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698107.302460:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698107.302463:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698107.302470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698107.302475:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126297344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698107.302483:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698107.302488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126297472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698107.302495:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698107.302500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126297600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698107.302511:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698107.302516:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126297728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698132 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698107.302522:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698107.813487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698107.813495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698107.813497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698107.941566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698108.837529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698108.837529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698108.837572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698108.965565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698109.221532:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300682607, at 4300682624 00000100:00000040:3.0:1787698109.221560:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300682607, at 4300682624 00000100:00100000:3.0:1787698109.861498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698109.861507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698109.861517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698109.989549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698110.885543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698110.885543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698110.885543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698111.013567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698111.909521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698111.909521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698111.909682:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698112.037562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698112.421590:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698112.421600:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698112.421604:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698112.421615:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6013 00000100:00080000:0.0:1787698112.421625:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698112.421629:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421633:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421636:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698112.421641:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698112.421644:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421647:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421650:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698112.421653:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421666:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698112.421691:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698112.421706:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698112.421751:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698112.421755:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698112.421757:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698112.421761:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6013 00000100:00080000:0.0:1787698112.421766:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698112.421769:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421772:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421774:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698112.421777:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698112.421781:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421784:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421787:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698112.421796:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698112.421806:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698112.421808:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698112.421816:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698112.421829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126297856:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698112.421831:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698112.421845:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698112.421855:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698112.421858:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698112.421862:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698112.421863:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698112.421867:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6013 00000100:00080000:0.0:1787698112.421872:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698112.421879:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421882:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421885:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421886:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698112.421888:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698112.421890:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421892:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421894:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698112.421896:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698112.421899:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698112.421903:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698112.421910:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698112.421916:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698112.421924:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698112.421924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698112.421926:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698112.421928:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698112.421932:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6013 00000100:00080000:0.0:1787698112.421936:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698112.421938:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421943:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698112.421945:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698112.421945:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698112.421950:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.421954:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.421957:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698112.421964:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698112.421968:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698112.421971:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698112.421978:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698112.421986:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.421989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698112/real 1787698112] req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698112.421994:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698112.422000:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698112.422002:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698112.422007:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6013 00000100:00080000:0.0:1787698112.422010:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698112.422013:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.422015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.422017:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698112.422019:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698112.422020:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698112.422023:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698112.422027:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698112.422030:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698112.422034:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698112.422038:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698112.422041:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698112.422044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698112.422048:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698112.422066:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698112.422066:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698112.422073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698112.422081:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6023) 00000100:00000040:1.0:1787698112.422083:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126297856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698112.422099:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126297856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698112.422109:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698112.422124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698112.422146:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422153:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698112.422167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698112.422174:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422184:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698112.422190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698112/real 1787698112] req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698112.422217:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698112.422221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698112.422224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126297984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698112.422238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698112.422257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422264:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698112.422274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698112.422281:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422289:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698112.422294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698112/real 1787698112] req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422318:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698112.422326:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698112.422329:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698112.422333:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422340:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698112.422346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698112.422366:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422373:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698112.422385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698112.422392:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422400:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698112.422406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698112/real 1787698112] req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422464:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698112.422472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698112.422475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698112.422478:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698112.422491:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698112.422511:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422519:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698112.422531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698112.422538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698137 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422547:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698112.422552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698112/real 1787698112] req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422562:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698112.422571:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698112.422578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698112.422581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698112.422584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698112.422592:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698112.422597:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126297984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698112.422605:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698112.422612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126298112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698112.422619:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698112.422624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126298240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698112.422631:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698112.422636:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126298368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698137 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698112.422642:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698112.933525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698112.933525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698112.933525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698113.061533:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698113.957499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698113.957500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698113.957590:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698114.085552:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698114.341519:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300687727, at 4300687744 00000100:00000040:3.0:1787698114.341519:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300687727, at 4300687744 00000100:00100000:3.0:1787698114.981498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698114.981500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698114.981501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698115.109547:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698116.005505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698116.005505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698116.005548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698116.133563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698117.029539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698117.029539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698117.029580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698117.157569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698117.541527:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698117.541537:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698117.541541:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698117.541552:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6018 00000100:00080000:0.0:1787698117.541562:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698117.541565:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541570:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541573:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698117.541577:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698117.541581:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541583:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541594:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698117.541597:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541609:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698117.541633:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698117.541647:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698117.541672:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698117.541675:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698117.541691:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698117.541696:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6018 00000100:00080000:0.0:1787698117.541701:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698117.541704:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541708:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541710:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698117.541713:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698117.541719:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698117.541720:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698117.541722:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541728:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698117.541734:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698117.541745:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698117.541748:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126298496:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698117.541752:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698117.541765:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698117.541777:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698117.541793:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698117.541808:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698117.541812:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698117.541814:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698117.541817:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698117.541822:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6018 00000100:00080000:0.0:1787698117.541836:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698117.541839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541842:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541844:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698117.541847:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:1.0:1787698117.541849:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541858:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541861:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541865:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787698117.541869:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698117.541871:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698117.541879:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698117.541885:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698117.541897:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698117.541907:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698117/real 1787698117] req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698117.541909:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698117.541935:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698117.541938:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698117.541941:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698117.541942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698117.541951:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6018 00000100:00080000:0.0:1787698117.541958:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698117.541961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.541967:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:1.0:1787698117.541969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698117.541971:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698117.541978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.541985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698117.541987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698117.541989:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698117.541994:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:1.0:1787698117.541997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:0.0:1787698117.542002:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698117.542005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698117.542008:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698117.542021:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698117.542022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126298496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698117.542033:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126298496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698117.542043:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698117.542049:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787698117.542054:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698117.542059:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698117.542061:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698117.542064:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6018 00000100:00080000:0.0:1787698117.542069:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698117.542070:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698117.542076:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.542083:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.542088:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698117.542090:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698117.542092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698117.542096:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698117.542101:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698117.542104:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698117.542106:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698117.542111:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698117.542113:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698117.542117:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698117.542127:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698117.542132:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698117.542142:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698117.542147:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698117.542159:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6028) 00000100:00000040:1.0:1787698117.542161:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542174:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698117.542181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698117/real 1787698117] req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698117.542207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698117.542211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698117.542215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698117.542227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542241:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698117.542252:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542260:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698117.542270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698117.542277:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542286:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698117.542292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698117/real 1787698117] req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698117.542318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698117.542322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698117.542326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542333:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698117.542338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698117.542375:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542382:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698117.542393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698117.542401:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542409:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698117.542415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698117/real 1787698117] req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542447:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542456:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698117.542463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698117.542466:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698117.542470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126298880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698117.542482:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126298624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698117.542489:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698117.542496:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126298752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698117.542507:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698117.542511:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126298880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698117.542518:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698117.542525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698117.542545:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542551:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698117.542563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698117.542570:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698142 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542579:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698117.542585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698117/real 1787698117] req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698117.542604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698117.542611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698117.542613:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698117.542617:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698117.542625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698117.542630:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126299008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698142 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698117.542637:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698118.053512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698118.053513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698118.053537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698118.181573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698119.077522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698119.077522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698119.077522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698119.205563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698119.461520:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300692847, at 4300692864 00000100:00000040:3.0:1787698119.461520:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300692847, at 4300692864 00000100:00100000:2.0:1787698120.101497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698120.101499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698120.101500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698120.229667:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698121.125516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698121.125517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698121.125517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698121.253556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698122.149468:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698122.149476:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698122.149544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698122.277541:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698122.661553:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698122.661562:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698122.661566:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698122.661575:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6023 00000100:00080000:0.0:1787698122.661584:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698122.661587:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661591:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661594:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698122.661599:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698122.661601:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661603:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661605:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698122.661607:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661617:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698122.661639:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698122.661654:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698122.661676:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698122.661680:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698122.661682:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698122.661687:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6023 00000100:00080000:0.0:1787698122.661690:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698122.661692:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661706:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661710:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:1.0:1787698122.661712:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698122.661713:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698122.661718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661724:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661727:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698122.661732:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698122.661737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126299136:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787698122.661739:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698122.661744:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698122.661753:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698122.661761:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698122.661763:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698122.661772:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698122.661779:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:1.0:1787698122.661779:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698122.661784:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698122.661789:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6023 00000100:00080000:0.0:1787698122.661802:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698122.661805:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661809:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698122.661811:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698122.661813:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661814:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661817:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698122.661819:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698122.661830:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698122.661835:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698122.661839:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698122.661847:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698122.661849:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698122.661865:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698122.661870:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698122.661886:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698122.661887:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698122.661892:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698122.661894:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698122.661899:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6023 00000100:00000040:1.0:1787698122.661899:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:0.0:1787698122.661907:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698122.661910:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698122.661913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661921:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661925:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698122.661929:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698122.661934:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698122.661936:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.661941:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661944:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698122.661949:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698122.661950:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698122.661956:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698122.661959:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698122.661964:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698122.661969:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698122.661977:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698122.661988:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698122.662018:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698122.662020:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698122.662021:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698122.662024:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6023 00000100:00080000:0.0:1787698122.662028:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698122.662030:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.662032:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.662034:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698122.662036:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698122.662039:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698122.662041:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698122.662047:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698122.662048:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698122.662050:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698122.662057:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698122.662061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299520:192.168.201.136@tcp:8:kworker.0 00000020:00000040:0.0:1787698122.662062:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698122.662075:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698122.662079:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698122.662089:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698122.662095:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698122.662104:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6033) 00000100:00000040:1.0:1787698122.662108:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787698122.662117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698122.662123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698122.662134:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698122.662140:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698122.662150:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698123.173551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698123.173552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698123.173558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698123.301598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698124.197557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698124.197557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698124.197560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698124.325611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698124.581524:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300697967, at 4300697984 00000100:00000040:2.0:1787698124.581540:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300697967, at 4300697984 00000100:00100000:3.0:1787698125.221529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698125.221562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698125.221562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698125.349623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698125.733748:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:2.0:1787698125.733810:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698125.733832:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787698125.733882:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698125.733904:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787698125.733918:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698125.733931:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787698125.733938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698122/real 1787698125] req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698125.733946:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698125.734003:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787698125.734023:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698125.734024:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698125.734100:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787698125.734113:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698147 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698125.734135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698125.734149:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698125.734155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698125.734160:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734168:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299264:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698125.734176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698122/real 1787698125] req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698125.734197:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698125.734204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698125.734208:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698125.734222:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126299136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698125.734238:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126299264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698125.734252:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698125.734262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126299136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698125.734269:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698125.734277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698122/real 1787698125] req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698125.734296:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698125.734304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698125.734307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698125.734311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299648:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698125.734325:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698122/real 1787698125] req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734335:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698125.734343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698125.734350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698125.734353:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698125.734357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734364:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299520:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698125.734370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698122/real 1787698125] req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698125.734386:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698125.734394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698125.734396:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698125.734400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698125.734406:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698125.734411:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126299648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698125.734417:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698125.734444:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126299520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698125.734454:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698125.734459:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126299392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698147 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698125.734473:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698126.245543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698126.245543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698126.245543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698126.373625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698127.269536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698127.269550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698127.269553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698127.397590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698127.781548:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698127.781557:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698127.781561:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698127.781572:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6028 00000100:00080000:0.0:1787698127.781584:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698127.781587:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781591:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781595:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698127.781599:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698127.781602:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781604:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781606:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698127.781608:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781620:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698127.781647:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698127.781660:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698127.781683:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698127.781685:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698127.781687:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698127.781690:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6028 00000100:00080000:0.0:1787698127.781692:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698127.781693:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781695:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781696:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698127.781716:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698127.781718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781722:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698127.781724:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698127.781731:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698127.781733:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698127.781741:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698127.781751:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126299776:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698127.781755:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698127.781767:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698127.781776:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698127.781780:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698127.781782:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698127.781786:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6028 00000100:00080000:0.0:1787698127.781790:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698127.781792:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698127.781796:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781803:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781809:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698127.781811:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698127.781813:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781817:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781821:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781823:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698127.781825:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698127.781831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698127.781836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698127.781851:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698127.781859:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698127.781867:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698127.781869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698127.781871:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698127.781874:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698127.781880:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6028 00000100:00080000:0.0:1787698127.781884:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698127.781888:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781892:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698127.781893:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698127.781896:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698127.781899:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698127.781905:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698127.781911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.781914:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698127.781917:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698127.781923:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698127.781930:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698127.781939:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698127.781945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698127/real 1787698127] req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698127.781955:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698127.781965:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698127.781970:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698127.781970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698127.781978:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698127.781985:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6028 00000100:00000040:1.0:1787698127.781989:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698127.781991:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698127.781998:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698127.782003:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698127.782004:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.782009:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:1.0:1787698127.782011:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698127.782016:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782024:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126299776:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698127.782026:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698127.782034:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698127.782035:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126299776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698127.782040:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698127.782045:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698127.782052:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698127.782061:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698127.782061:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:0.0:1787698127.782069:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698127.782080:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698127.782089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698127.782093:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698127.782115:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6038) 00000100:00000040:1.0:1787698127.782130:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782140:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698127.782155:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698127.782162:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782186:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698127.782192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698127/real 1787698127] req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782213:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698127.782221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698127.782224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698127.782228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782236:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126299904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698127.782244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698127.782266:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782273:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698127.782285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698127.782292:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782301:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698127.782307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698127/real 1787698127] req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698127.782333:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698127.782336:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698127.782339:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698127.782351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698127.782374:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782382:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698127.782392:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698127.782399:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782409:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698127.782415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698127/real 1787698127] req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782464:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698127.782470:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698127.782473:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698127.782476:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782483:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698127.782489:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126299904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698127.782496:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698127.782503:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126300032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698127.782510:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698127.782514:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126300160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698127.782521:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698127.782529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698127.782547:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782551:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698127.782563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698127.782569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698152 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782585:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698127.782591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698127/real 1787698127] req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698127.782610:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698127.782618:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698127.782621:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698127.782625:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698127.782634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698127.782640:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126300288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698152 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698127.782646:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698128.293537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698128.293537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698128.293537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698128.421576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698129.317491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698129.317492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698129.317525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698129.445640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698129.701562:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300703087, at 4300703104 00000100:00000040:2.0:1787698129.701655:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300703087, at 4300703104 00000100:00100000:2.0:1787698130.341607:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698130.341612:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698130.341649:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698130.469607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698131.365512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698131.365512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698131.365515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698131.493624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698132.389511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698132.389511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698132.389578:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698132.517533:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698132.901500:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698132.901510:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698132.901515:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698132.901525:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6033 00000100:00080000:0.0:1787698132.901536:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698132.901539:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901543:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901562:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698132.901566:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698132.901570:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901575:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698132.901579:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901589:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698132.901610:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698132.901625:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.901646:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698132.901649:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698132.901651:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698132.901655:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6033 00000100:00080000:0.0:1787698132.901668:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698132.901671:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901675:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901677:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698132.901680:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698132.901683:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901685:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901689:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787698132.901691:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.901693:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698132.901706:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698132.901711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126300416:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698132.901713:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698132.901735:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698132.901744:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698132.901748:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.901769:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698132.901776:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698132.901781:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698132.901783:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698132.901787:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6033 00000100:00080000:0.0:1787698132.901792:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698132.901795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901800:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698132.901803:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:1.0:1787698132.901807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901808:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901814:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901816:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698132.901820:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698132.901825:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698132.901827:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698132.901831:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698132.901843:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698132.901852:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.901861:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698132.901865:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787698132.901865:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698132/real 1787698132] req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.901874:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698132.901878:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.901881:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6033 00000100:00000040:1.0:1787698132.901889:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.901897:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698132.901904:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698132.901905:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698132.901912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698132.901915:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787698132.901916:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698132.901920:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698132.901922:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698132.901926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901932:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.901937:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:1.0:1787698132.901941:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126300416:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698132.901942:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698132.901951:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698132.901956:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126300416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698132.901957:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698132.901971:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787698132.901971:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698132.901984:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.901988:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.902006:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698132.902007:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300544:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698132.902015:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698132.902018:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698132.902023:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6033 00000100:00000040:1.0:1787698132.902025:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698132.902028:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698132.902034:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698132.902037:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.902041:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787698132.902042:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698132.902046:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:1.0:1787698132.902055:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698132.902062:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698132.902064:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698132.902070:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698132.902075:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698132.902077:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:0.0:1787698132.902080:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698132.902084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698132/real 1787698132] req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698132.902103:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698132.902111:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698132.902117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698132.902124:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698132.902151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698132.902153:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698132.902166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698132.902173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698132.902175:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6044) 00000100:00000040:1.0:1787698132.902177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698132.902186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698132.902206:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902210:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698132.902218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698132.902222:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902228:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698132.902231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698132/real 1787698132] req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698132.902260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698132.902263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698132.902268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698132.902282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126300544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698132.902287:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698132.902291:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126300672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698132.902295:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698132.902300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698132.902315:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902319:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698132.902326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698132.902330:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698132.902347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698132/real 1787698132] req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698132.902367:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698132.902369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698132.902371:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902375:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698132.902378:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698132.902389:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902394:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698132.902399:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698132.902403:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698157 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902413:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698132.902416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698132/real 1787698132] req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902443:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698132.902449:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698132.902453:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698132.902455:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698132.902457:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698132.902461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126300928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698132.902464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126300800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698132.902473:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698132.902476:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126300928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698157 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698132.902480:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698133.413490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698133.413497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698133.413499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787698133.541592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698134.437516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698134.437516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698134.437516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698134.565598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698134.821507:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300708207, at 4300708224 00000100:00000040:3.0:1787698134.821507:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300708207, at 4300708224 00000100:00100000:1.0:1787698135.461504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698135.461504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698135.461504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698135.589542:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698136.485521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698136.485521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698136.485521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787698136.613592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698137.509576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698137.509706:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698137.509794:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698137.637630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698138.021530:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.021540:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698138.021544:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.021554:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6039 00000100:00080000:0.0:1787698138.021564:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698138.021567:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021571:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698138.021574:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698138.021579:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698138.021582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021585:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698138.021588:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698138.021590:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698138.021602:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698138.021620:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698138.021642:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698138.021656:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.021659:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698138.021660:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.021671:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6039 00000100:00080000:0.0:1787698138.021676:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698138.021679:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021684:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698138.021685:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698138.021687:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698138.021692:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698138.021699:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021703:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698138.021705:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00100000:1.0:1787698138.021708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126301056:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698138.021711:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698138.021723:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698138.021729:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698138.021744:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698138.021744:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.021761:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021775:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698138.021785:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.021788:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698138.021790:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698138.021801:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698138.021804:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6039 00000100:00080000:0.0:1787698138.021810:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698138.021813:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021817:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698138.021820:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787698138.021822:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698138.021823:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698138.021829:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021835:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698138.021839:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698138.021843:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698138.021848:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698138.021852:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698138.021861:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698138.021872:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.021878:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698138.021883:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.021890:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698138.021893:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.021899:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6039 00000100:00080000:0.0:1787698138.021904:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698138.021906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698138.021909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698138.021919:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698138.021923:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698138.021928:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698138.021929:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.021940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698138.021940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698138.021944:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698138.021948:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:1.0:1787698138.021949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:0.0:1787698138.021953:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698138.021958:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698138.021959:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698138.021971:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126301056:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698138.021973:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698138.021983:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126301056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698138.021990:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698138.022010:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787698138.022013:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698138.022017:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698138.022019:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698138.022022:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6039 00000100:00080000:0.0:1787698138.022027:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698138.022029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698138.022033:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.022040:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698138.022043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301184:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787698138.022044:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698138.022049:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698138.022052:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698138.022054:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698138.022055:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698138.022057:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698138.022060:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698138.022062:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698138.022068:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698138.022076:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698138.022082:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698138.022089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698138.022097:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698138.022106:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022107:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698138.022115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698138.022128:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6049) 00000100:00100000:1.0:1787698138.022129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.022142:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.022144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698138.022147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022153:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698138.022156:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698138.022167:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022171:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698138.022187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698138.022193:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022201:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698138.022207:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.022232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.022234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698138.022236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022240:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698138.022243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698138.022256:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022261:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698138.022270:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698138.022276:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022290:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698138.022295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.022328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.022331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698138.022335:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698138.022347:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126301184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698138.022355:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698138.022361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126301312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698138.022367:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698138.022372:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126301440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698138.022386:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698138.022394:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698138.022414:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022437:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698138.022454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698138.022459:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698163 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022464:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698138.022468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698138/real 1787698138] req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698138.022479:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698138.022483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698138.022485:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698138.022487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698138.022491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698138.022494:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126301568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698163 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698138.022498:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698138.533524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698138.533524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698138.533530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698138.661581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698139.557514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698139.557514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698139.557514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698139.685603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698139.941512:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300713327, at 4300713344 00000100:00000040:2.0:1787698139.941513:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300713327, at 4300713344 00000100:00100000:0.0:1787698140.581529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698140.581529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698140.581529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698140.709590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698141.605495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698141.605495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698141.605499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698141.733585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698142.629530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698142.629530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698142.629535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698142.757568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698143.141523:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698143.141532:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698143.141537:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698143.141546:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6044 00000100:00080000:3.0:1787698143.141555:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698143.141558:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141562:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141565:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698143.141570:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698143.141573:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141575:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141578:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698143.141581:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141593:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698143.141618:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698143.141633:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698143.141653:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698143.141657:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698143.141659:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698143.141663:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6044 00000100:00080000:3.0:1787698143.141667:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698143.141670:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141686:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141690:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:1.0:1787698143.141693:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698143.141693:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698143.141700:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141706:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141710:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698143.141714:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698143.141718:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126301696:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698143.141724:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698143.141731:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698143.141741:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698143.141745:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698143.141753:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141763:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698143.141773:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698143.141776:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698143.141778:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698143.141782:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6044 00000800:00000100:1.0:1787698143.141785:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698143.141789:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698143.141795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698143.141801:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698143.141803:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698143.141807:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698143.141813:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141817:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141819:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698143.141822:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698143.141826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698143.141837:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698143.141841:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698143.141845:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698143.141852:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.141858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698143.141861:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698143.141862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698143.141868:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.141885:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126301696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698143.141889:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.141893:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126301696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698143.141928:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698143.141928:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698143.141934:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698143.141936:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698143.141941:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6044 00000100:00000040:1.0:1787698143.141944:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698143.141945:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698143.141953:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698143.141957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698143.141960:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141964:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698143.141967:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698143.141971:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698143.141972:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698143.141979:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698143.141985:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698143.141985:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698143.141991:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:1.0:1787698143.141997:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698143.141997:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698143.142003:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698143.142004:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698143.142012:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698143.142018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698143.142026:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698143.142027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698143.142055:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.142058:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.142064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698143.142070:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142075:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698143.142084:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698143.142098:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698143.142103:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698143.142105:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698143.142109:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698143.142112:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698143.142113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698143.142121:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6044 00000100:00000040:1.0:1787698143.142122:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698143.142130:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698143.142131:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787698143.142137:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698143.142138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698143.142145:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698143.142149:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:1.0:1787698143.142150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698143.142153:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698143.142161:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698143.142164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698143.142167:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698143.142175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.142177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698143.142180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698143.142180:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698143.142187:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698143.142192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126301952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698143.142197:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126301824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698143.142197:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698143.142207:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000020:00000040:3.0:1787698143.142207:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698143.142215:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126301952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698143.142220:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698143.142224:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698143.142228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787698143.142250:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142267:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698143.142268:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6054) 00000800:00000100:1.0:1787698143.142274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698143.142279:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142286:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698143.142289:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142296:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.142305:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.142307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698143.142309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142313:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698143.142315:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126302080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698143.142319:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698143.142323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698143.142333:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142337:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698143.142341:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698143.142345:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698168 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142350:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698143.142352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698143/real 1787698143] req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698143.142367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698143.142371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698143.142372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698143.142374:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698143.142378:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698143.142381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126302208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698168 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698143.142384:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698143.653476:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698143.653476:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698143.653489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698143.781560:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698144.677521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698144.677521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698144.677524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698144.805538:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698145.061500:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300718447, at 4300718464 00000100:00000040:3.0:1787698145.061500:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300718447, at 4300718464 00000100:00100000:2.0:1787698145.701528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698145.701528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698145.701532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698145.829581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698146.725509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698146.725509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698146.725509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698146.853573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698147.749525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698147.749525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698147.749614:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698147.877580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698148.261534:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698148.261544:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698148.261548:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698148.261557:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6049 00000100:00080000:1.0:1787698148.261566:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698148.261570:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261574:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261577:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698148.261582:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698148.261585:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261588:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261590:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698148.261593:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261605:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698148.261626:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698148.261651:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698148.261671:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698148.261675:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698148.261677:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698148.261690:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6049 00000100:00000040:0.0:1787698148.261695:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698148.261696:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698148.261703:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261709:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698148.261713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126302336:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698148.261714:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698148.261719:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698148.261724:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261727:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261729:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698148.261732:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698148.261739:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698148.261742:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698148.261749:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261751:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698148.261764:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698148.261775:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698148.261783:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698148.261790:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698148.261793:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698148.261796:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698148.261801:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698148.261803:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6049 00000100:00080000:1.0:1787698148.261809:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698148.261814:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261817:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261819:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698148.261821:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698148.261840:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261844:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261846:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698148.261849:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698148.261854:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698148.261860:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698148.261871:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698148.261874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698148/real 1787698148] req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698148.261890:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698148.261897:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698148.261900:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787698148.261902:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698148.261903:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698148.261907:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6049 00000100:00080000:1.0:1787698148.261910:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698148.261912:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261917:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261920:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698148.261923:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698148.261927:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.261930:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.261932:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698148.261937:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698148.261939:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698148.261942:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698148.261949:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698148.261954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698148.261961:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698148.261976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698148.261977:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.261983:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698148.261987:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698148.261997:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787698148.261999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126302336:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698148.262000:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698148.262008:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126302336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698148.262008:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6049 00000100:00080000:1.0:1787698148.262015:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000020:00000040:0.0:1787698148.262020:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698148.262021:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698148.262026:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.262029:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698148.262032:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698148.262035:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698148.262038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698148.262039:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698148.262044:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698148.262048:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698148.262054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302464:192.168.201.136@tcp:38:kworker.0 00000008:00100000:1.0:1787698148.262054:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698148.262062:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698148.262072:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698148.262073:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262094:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698148.262094:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698148.262105:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6059) 00000800:00000100:0.0:1787698148.262110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698148.262117:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262125:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698148.262128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698148/real 1787698148] req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262140:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698148.262152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698148.262154:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698148.262157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698148.262164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698148.262176:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262180:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698148.262186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698148.262190:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262196:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698148.262199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698148/real 1787698148] req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698148.262223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698148.262229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698148.262231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698148.262239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698148.262251:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262255:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698148.262261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698148.262265:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262271:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698148.262274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698148/real 1787698148] req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698148.262289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698148.262291:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698148.262293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698148.262304:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126302464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698148.262309:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698148.262312:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126302592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698148.262316:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698148.262319:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126302720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698148.262323:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698148.262328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698148.262339:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262343:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698148.262349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698148.262353:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698173 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262358:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698148.262361:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698148/real 1787698148] req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698148.262380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698148.262384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698148.262386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698148.262388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698148.262391:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126302848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698148.262394:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126302848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698173 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698148.262398:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698148.773466:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698148.773555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698148.773573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698148.901615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698149.797520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698149.797520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698149.797545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698149.925555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698150.181521:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300723567, at 4300723584 00000100:00000040:3.0:1787698150.181521:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300723567, at 4300723584 00000100:00100000:1.0:1787698150.821528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698150.821528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698150.821528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698150.949596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698151.845585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698151.845586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698151.845586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698151.973577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698152.869535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698152.869535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698152.869535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698152.997570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698153.381535:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698153.381543:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698153.381546:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698153.381554:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6054 00000100:00080000:3.0:1787698153.381563:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698153.381565:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698153.381571:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381574:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698153.381578:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698153.381580:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698153.381582:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381584:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698153.381586:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381597:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698153.381615:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698153.381625:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698153.381639:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698153.381641:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698153.381654:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698153.381658:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6054 00000100:00080000:3.0:1787698153.381661:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698153.381663:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698153.381665:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381667:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698153.381669:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698153.381674:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698153.381677:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698153.381678:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698153.381679:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698153.381683:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698153.381693:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698153.381697:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698153.381699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126302976:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698153.381706:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698153.381713:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698153.381720:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698153.381721:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698153.381722:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698153.381725:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6054 00000100:00080000:3.0:1787698153.381727:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698153.381731:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698153.381734:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698153.381735:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698153.381758:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381762:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698153.381765:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698153.381776:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698153.381778:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381779:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698153.381781:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698153.381785:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698153.381788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:0.0:1787698153.381796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698153.381796:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698153.381808:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.381816:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698153.381819:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698153.381826:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698153.381830:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698153.381836:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6054 00000100:00080000:3.0:1787698153.381841:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698153.381844:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698153.381847:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381849:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698153.381852:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698153.381855:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698153.381860:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698153/real 1787698153] req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698153.381860:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381862:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698153.381864:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698153.381868:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698153.381874:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698153.381884:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698153.381893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698153.381901:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698153.381916:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787698153.381920:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698153.381920:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698153.381923:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698153.381929:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6054 00000100:00080000:3.0:1787698153.381935:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698153.381939:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698153.381940:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698153.381942:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381946:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698153.381948:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:0.0:1787698153.381949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698153.381952:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698153.381955:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698153.381959:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787698153.381960:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698153.381962:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698153.381967:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698153.381972:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698153.381979:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126302976:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698153.381981:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698153.381993:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126302976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698153.381993:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698153.382006:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698153.382009:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6064) 00000100:00000040:0.0:1787698153.382025:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698153.382049:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382057:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698153.382069:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698153.382075:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382086:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698153.382091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698153/real 1787698153] req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382109:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698153.382125:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698153.382128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698153.382131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698153.382144:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382152:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698153.382162:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382169:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698153.382179:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698153.382186:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382194:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698153.382199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698153/real 1787698153] req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698153.382237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698153.382240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698153.382244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698153.382256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698153.382276:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382283:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698153.382295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698153.382302:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382321:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698153.382327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698153/real 1787698153] req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698153.382363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698153.382366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698153.382370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698153.382383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698153.382403:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382410:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698153.382440:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698153.382448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698178 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382457:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698153.382462:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698153/real 1787698153] req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698153.382487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698153.382494:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698153.382497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698153.382501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698153.382508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698153.382514:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126303104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698153.382523:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698153.382528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126303232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698153.382535:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698153.382539:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126303360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698153.382546:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698153.382550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126303488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698178 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698153.382557:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698153.893520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698153.893520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698153.893520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698154.021554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698154.917565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698154.917566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698154.917570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698155.045617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698155.301578:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300728687, at 4300728704 00000100:00000040:3.0:1787698155.301578:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300728687, at 4300728704 00000100:00100000:3.0:1787698155.941573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698155.941574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698155.941606:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698156.069599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698156.965491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698156.965492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698156.965520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698157.093620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698157.989552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698157.989553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698157.989553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698158.117595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698158.501553:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698158.501566:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698158.501572:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698158.501585:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6059 00000100:00080000:3.0:1787698158.501599:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698158.501603:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.501607:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501611:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698158.501616:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698158.501620:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.501622:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501626:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698158.501629:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501647:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698158.501708:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698158.501723:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698158.501749:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698158.501752:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698158.501754:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698158.501758:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6059 00000100:00080000:3.0:1787698158.501762:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698158.501782:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.501785:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501788:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698158.501791:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698158.501796:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698158.501799:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698158.501800:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501803:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698158.501806:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698158.501816:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698158.501824:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698158.501834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126303616:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698158.501838:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698158.501849:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698158.501859:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698158.501862:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698158.501864:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698158.501868:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6059 00000100:00000040:0.0:1787698158.501873:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698158.501874:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698158.501882:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.501887:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698158.501890:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501891:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698158.501895:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698158.501899:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.501903:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501904:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698158.501913:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698158.501919:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698158.501924:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:0.0:1787698158.501931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698158.501937:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698158.501948:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.501957:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698158.501958:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698158.501967:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698158.501970:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698158.501974:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6059 00000100:00080000:3.0:1787698158.501978:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698158.501981:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.501984:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501986:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698158.501988:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698158.501991:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.501994:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.501997:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698158.502000:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698158.502003:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698158/real 1787698158] req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698158.502017:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698158.502023:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698158.502024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698158.502036:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698158.502040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698158.502058:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698158.502060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698158.502072:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698158.502075:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698158.502077:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698158.502081:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698158.502084:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698158.502088:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6059 00000100:00080000:3.0:1787698158.502094:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698158.502100:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.502103:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.502106:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:0.0:1787698158.502109:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126303616:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698158.502110:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698158.502116:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698158.502120:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698158.502123:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787698158.502124:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126303616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698158.502128:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698158.502137:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698158.502140:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:3.0:1787698158.502143:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698158.502154:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698158.502169:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698158.502197:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6069) 00000100:00100000:0.0:1787698158.502209:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698158.502222:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502228:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698158.502238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698158.502246:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502256:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698158.502262:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698158/real 1787698158] req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698158.502288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698158.502290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698158.502293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698158.502301:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698158.502326:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502332:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698158.502343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698158.502351:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502369:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698158.502375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698158/real 1787698158] req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698158.502395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698158.502397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698158.502399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126303872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698158.502407:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698158.502444:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502453:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698158.502462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698158.502467:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502473:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698158.502476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698158/real 1787698158] req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698158.502493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698158.502504:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698158.502508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698158.502521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126303744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698158.502529:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698158.502535:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126303872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698158.502541:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698158.502546:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126304000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698158.502559:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698158.502567:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698158.502588:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502595:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698158.502607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698158.502614:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698183 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502623:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698158.502629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698158/real 1787698158] req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698158.502666:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698158.502674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698158.502676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698158.502679:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698158.502683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698158.502693:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126304128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698183 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698158.502698:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698159.013492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698159.013513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698159.013562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698159.141629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698160.037546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698160.037546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698160.037546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698160.165578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698160.421552:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300733807, at 4300733824 00000100:00000040:3.0:1787698160.421552:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300733807, at 4300733824 00000100:00100000:0.0:1787698161.061555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698161.061555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698161.061555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698161.189589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698162.085538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698162.085539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698162.085539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698162.213566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698163.109563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698163.109567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698163.109605:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698163.237596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698163.621549:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698163.621571:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698163.621577:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698163.621591:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6064 00000100:00080000:3.0:1787698163.621604:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698163.621608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698163.621613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621617:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698163.621623:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698163.621627:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698163.621635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621639:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698163.621642:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621661:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698163.621691:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698163.621707:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698163.621734:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698163.621738:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698163.621740:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698163.621745:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6064 00000100:00080000:3.0:1787698163.621749:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698163.621752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698163.621755:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698163.621777:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698163.621780:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698163.621785:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698163.621792:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698163.621795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126304256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698163.621797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621803:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698163.621807:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698163.621819:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698163.621825:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698163.621836:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698163.621847:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698163.621857:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698163.621860:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698163.621862:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698163.621866:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6064 00000100:00080000:3.0:1787698163.621870:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698163.621873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698163.621878:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621880:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698163.621883:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698163.621886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698163.621889:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621891:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698163.621894:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698163.621904:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698163.621905:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698163.621919:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621920:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698163.621935:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698163.621947:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698163.621957:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698163.621961:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698163.621962:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698163.621968:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6064 00000800:00000100:0.0:1787698163.621972:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698163.621975:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698163.621981:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698163.621984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.621986:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698163.621989:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698163.621993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698163.621998:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698163.621999:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698163.622003:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698163.622010:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698163.622016:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698163.622020:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698163.622031:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698163.622041:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622050:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698163/real 1787698163] req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698163.622051:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698163.622063:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698163.622066:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698163.622072:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6064 00000100:00100000:0.0:1787698163.622073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698163.622078:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698163.622085:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698163.622091:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698163.622092:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.622098:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698163.622102:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:0.0:1787698163.622106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698163.622107:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698163.622115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698163.622116:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698163.622119:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698163.622123:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698163.622130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698163.622131:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698163.622139:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698163.622144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126304256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698163.622152:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126304256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698163.622155:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787698163.622169:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698163.622174:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698163.622187:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6074) 00000100:00000040:0.0:1787698163.622193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622204:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698163.622219:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622227:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698163.622238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698163.622244:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622262:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698163.622268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698163/real 1787698163] req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698163.622295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698163.622303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698163.622305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698163.622314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698163.622327:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622332:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698163.622340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698163.622344:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622350:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698163.622353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698163/real 1787698163] req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622365:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698163.622369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698163.622370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698163.622373:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698163.622385:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698163.622399:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622404:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698163.622412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698163.622447:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622458:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698163.622463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698163/real 1787698163] req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622477:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698163.622481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698163.622482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698163.622484:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622488:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698163.622496:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622502:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698163.622511:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622515:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698163.622522:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698163.622526:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698188 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622532:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698163.622535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698163/real 1787698163] req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698163.622547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698163.622551:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698163.622552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698163.622554:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698163.622558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126304768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698163.622599:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126304384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698163.622619:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698163.622624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126304512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698163.622630:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698163.622636:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126304640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698163.622643:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698163.622648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126304768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698188 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698163.622655:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698164.133565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698164.133569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698164.133663:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698164.261722:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698165.157568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698165.157575:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698165.157583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698165.285604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698165.541547:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300738927, at 4300738944 00000100:00000040:3.0:1787698165.541547:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300738927, at 4300738944 00000100:00100000:2.0:1787698166.181563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698166.181575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698166.181578:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698166.309615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698167.205545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698167.205574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698167.205576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698167.333588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698168.229549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698168.229549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698168.229564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698168.357567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698168.741601:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698168.741608:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698168.741612:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698168.741620:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6069 00000100:00080000:1.0:1787698168.741630:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698168.741635:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741640:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741643:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698168.741647:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698168.741650:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741652:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741654:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698168.741656:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741668:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698168.741687:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698168.741699:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698168.741718:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698168.741720:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698168.741721:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698168.741724:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6069 00000100:00080000:1.0:1787698168.741726:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698168.741727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741730:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698168.741732:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698168.741734:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741736:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698168.741738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698168.741744:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698168.741747:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698168.741754:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698168.741760:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698168.741768:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787698168.741771:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698168.741772:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698168.741775:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698168.741784:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6069 00000100:00080000:1.0:1787698168.741790:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698168.741793:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741796:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741798:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698168.741800:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698168.741803:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741809:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698168.741813:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698168.741814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126304896:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787698168.741817:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698168.741823:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698168.741830:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698168.741837:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698168.741842:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698168.741843:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698168.741845:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698168.741850:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6069 00000100:00000040:0.0:1787698168.741851:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698168.741853:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698168.741859:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741865:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698168.741869:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741871:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698168.741874:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698168.741878:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741884:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698168.741887:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698168.741892:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698168.741899:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000800:00000100:0.0:1787698168.741903:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698168.741912:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698168.741923:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698168.741925:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698168.741950:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698168.741953:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698168.741954:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698168.741958:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6069 00000100:00080000:1.0:1787698168.741961:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698168.741963:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741966:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741967:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698168.741969:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:0.0:1787698168.741972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698168/real 1787698168] req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698168.741974:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698168.741976:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698168.741977:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698168.741979:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698168.741982:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698168.741987:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698168.741997:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698168.742003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698168.742009:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742022:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698168.742023:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6079) 00000100:00080000:0.0:1787698168.742031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698168.742034:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698168.742038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126304896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698168.742048:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126304896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698168.742055:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698168.742067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742072:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698168.742082:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742087:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698168.742096:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698168.742100:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742107:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698168.742111:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698168/real 1787698168] req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698168.742133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698168.742134:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698168.742136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698168.742147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698168.742166:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742174:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698168.742185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698168.742192:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742201:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698168.742206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698168/real 1787698168] req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698168.742233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698168.742235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698168.742237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698168.742252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698168.742269:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742274:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698168.742282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698168.742286:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742292:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698168.742295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698168/real 1787698168] req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698168.742331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698168.742334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698168.742338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698168.742351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698168.742372:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742376:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698168.742384:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698168.742389:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698193 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742397:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698168.742402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698168/real 1787698168] req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742412:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698168.742456:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698168.742461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698168.742463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698168.742465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698168.742470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698168.742484:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126305024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698168.742490:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698168.742494:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126305152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698168.742499:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698168.742501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126305280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698168.742506:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698168.742510:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126305408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698193 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698168.742516:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698169.253540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698169.253545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698169.253556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698169.381557:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698170.277560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698170.277560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698170.277563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698170.405643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698170.661587:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300744047, at 4300744064 00000100:00000040:3.0:1787698170.661587:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300744047, at 4300744064 00000100:00100000:0.0:1787698171.301550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698171.301550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698171.301602:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698171.429529:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698172.325530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698172.325548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698172.325648:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698172.453612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698173.349543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698173.349543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698173.349547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698173.477569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698173.861566:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698173.861577:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698173.861582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698173.861593:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6074 00000100:00080000:2.0:1787698173.861604:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698173.861606:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861613:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698173.861618:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698173.861620:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861624:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698173.861627:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861642:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698173.861667:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698173.861686:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698173.861707:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698173.861710:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698173.861711:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698173.861714:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6074 00000100:00080000:2.0:1787698173.861716:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698173.861717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861719:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861720:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698173.861722:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698173.861724:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861726:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861728:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698173.861729:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698173.861741:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698173.861744:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698173.861753:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698173.861756:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698173.861762:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698173.861785:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698173.861788:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698173.861790:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698173.861792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126305536:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698173.861795:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6074 00000100:00080000:2.0:1787698173.861799:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698173.861801:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861803:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861804:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698173.861806:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698173.861808:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861809:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861811:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698173.861812:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698173.861815:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698173.861819:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698173.861824:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698173.861829:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698173.861835:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.861847:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698173.861852:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698173.861855:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698173.861857:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698173.861862:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6074 00000100:00080000:2.0:1787698173.861865:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698173.861868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861871:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861874:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698173.861876:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698173.861879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861885:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698173.861888:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698173.861893:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787698173.861899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861900:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698173.861912:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698173.861919:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.861925:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698173.861927:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698173.861933:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698173.861936:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698173.861943:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6074 00000100:00080000:2.0:1787698173.861953:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698173.861955:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861956:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861958:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698173.861959:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698173.861961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698173.861964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698173.861965:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698173.861967:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698173.861970:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698173.861974:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698173.861979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698173/real 1787698173] req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698173.861984:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698173.861999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698173.862001:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698173.862017:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6084) 00000100:00000040:0.0:1787698173.862024:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698173.862035:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698173.862039:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698173.862043:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862056:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126305536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698173.862063:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126305536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698173.862073:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698173.862109:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862117:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698173.862131:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862138:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698173.862151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698173.862165:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862176:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698173.862180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698173/real 1787698173] req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698173.862197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698173.862199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698173.862201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698173.862209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698173.862237:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862242:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698173.862252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698173.862257:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862263:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698173.862266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698173/real 1787698173] req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862273:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862279:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698173.862284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698173.862285:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698173.862287:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698173.862294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698173.862313:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862318:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698173.862324:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698173.862328:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862334:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698173.862337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698173/real 1787698173] req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698173.862354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698173.862355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698173.862357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126305920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698173.862364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698173.862374:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862397:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698173.862409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698173.862416:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698198 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862450:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698173.862455:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698173/real 1787698173] req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698173.862468:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698173.862472:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698173.862474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698173.862476:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698173.862480:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698173.862484:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126305664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698173.862489:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698173.862493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126305792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698173.862497:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698173.862501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126305920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698173.862510:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698173.862513:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126306048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698198 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698173.862517:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698174.373539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698174.373539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698174.373544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698174.501630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698175.397575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698175.397576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698175.397582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698175.525611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698175.781555:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300749167, at 4300749184 00000100:00000040:3.0:1787698175.781555:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300749167, at 4300749184 00000100:00100000:1.0:1787698176.421502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698176.421502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698176.421506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698176.549649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698177.445546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698177.445546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698177.445546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698177.573637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698178.469507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698178.469507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698178.469514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698178.597627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698178.981546:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698178.981563:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698178.981568:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698178.981594:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6079 00000100:00080000:3.0:1787698178.981609:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698178.981612:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981618:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.981622:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698178.981628:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698178.981632:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.981639:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698178.981642:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698178.981661:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698178.981695:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698178.981712:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698178.981734:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698178.981738:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698178.981740:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698178.981746:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6079 00000100:00080000:3.0:1787698178.981751:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698178.981765:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698178.981776:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981780:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698178.981783:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126306176:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698178.981785:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698178.981789:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698178.981795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.981800:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698178.981803:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698178.981813:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698178.981819:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698178.981824:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698178.981835:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698178.981839:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981851:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698178.981861:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698178.981865:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698178.981867:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698178.981872:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6079 00000100:00080000:3.0:1787698178.981876:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698178.981879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.981885:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000800:00000100:0.0:1787698178.981888:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698178.981889:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698178.981895:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981899:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.981901:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698178.981910:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698178.981913:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698178.981917:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698178.981925:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698178.981938:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698178.981950:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.981959:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698178/real 1787698178] req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698178.981960:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698178.981969:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698178.981972:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698178.981981:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6079 00000100:00080000:3.0:1787698178.981987:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698178.981989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698178.981993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.981999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.982005:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698178.982009:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698178.982011:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698178.982016:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.982024:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.982028:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698178.982031:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698178.982036:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698178.982046:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698178.982050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698178.982058:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:3.0:1787698178.982060:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698178.982067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698178.982078:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982083:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126306176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698178.982098:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126306176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698178.982098:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698178.982106:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698178.982110:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698178.982118:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6079 00000020:00000040:0.0:1787698178.982121:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698178.982123:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698178.982128:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.982132:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698178.982134:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698178.982137:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698178.982140:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698178.982144:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698178.982147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698178.982148:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698178.982155:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698178.982165:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787698178.982169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306304:192.168.201.136@tcp:38:kworker.0 00000020:00000040:3.0:1787698178.982172:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698178.982187:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698178.982191:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698178.982213:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982214:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698178.982228:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6090) 00000800:00000100:0.0:1787698178.982230:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698178.982251:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982264:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698178.982271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698178/real 1787698178] req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698178.982329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698178.982333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698178.982337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698178.982351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698178.982373:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982381:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698178.982391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698178.982398:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982408:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698178.982414:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698178/real 1787698178] req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982454:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698178.982471:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698178.982474:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698178.982478:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982485:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698178.982490:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982498:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698178.982512:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982519:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698178.982531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698178.982538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982548:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698178.982553:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698178/real 1787698178] req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982565:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982578:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698178.982586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698178.982594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698178.982599:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982606:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698178.982612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126306304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698178.982619:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698178.982626:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126306432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698178.982633:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698178.982638:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126306560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698178.982645:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698178.982653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982659:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698178.982668:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982673:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698178.982681:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698178.982685:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698203 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982691:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698178.982695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698178/real 1787698178] req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982706:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698178.982712:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698178.982716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698178.982718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698178.982720:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698178.982724:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698178.982728:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126306688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698203 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698178.982733:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698179.493557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698179.493557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698179.493557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698179.621602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698180.517532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698180.517536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698180.517587:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698180.645622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698180.901515:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300754287, at 4300754304 00000100:00000040:3.0:1787698180.901515:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300754287, at 4300754304 00000100:00100000:0.0:1787698181.541500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698181.541500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698181.541517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698181.669568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698182.565515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698182.565516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698182.565516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698182.693577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698183.589525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698183.589526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698183.589542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698183.717570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698184.101515:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.101522:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698184.101526:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.101533:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6085 00000100:00080000:0.0:1787698184.101541:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698184.101543:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101563:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698184.101567:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698184.101569:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101570:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101572:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698184.101574:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101583:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698184.101601:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698184.101612:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698184.101630:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.101632:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698184.101634:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.101636:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6085 00000100:00080000:0.0:1787698184.101638:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698184.101640:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101641:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101643:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698184.101645:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698184.101646:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101648:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101649:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698184.101650:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698184.101664:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698184.101671:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698184.101678:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698184.101685:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698184.101689:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698184.101711:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.101714:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698184.101716:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698184.101717:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126306816:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698184.101720:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6085 00000100:00080000:0.0:1787698184.101724:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698184.101727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101730:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698184.101731:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698184.101733:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101736:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698184.101738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698184.101741:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698184.101744:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698184.101749:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698184.101753:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698184.101760:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.101772:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698184.101773:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.101780:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698184.101782:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.101787:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6085 00000100:00080000:0.0:1787698184.101808:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698184.101812:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101815:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101817:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698184.101819:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:1.0:1787698184.101824:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101825:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101830:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101833:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698184.101836:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698184.101840:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698184.101845:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698184.101845:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698184.101855:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698184.101862:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698184.101867:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698184.101869:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698184.101870:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698184.101873:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6085 00000100:00080000:0.0:1787698184.101876:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698184.101880:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698184.101881:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698184.101883:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101891:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698184.101894:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698184.101897:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698184.101899:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698184.101900:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698184.101904:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698184.101906:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698184.101908:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698184.101913:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698184.101924:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698184.101926:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698184.101945:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.101945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.101953:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698184.101961:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698184.101962:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6095) 00000100:00100000:1.0:1787698184.101972:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126306816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698184.101979:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126306816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698184.101994:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698184.102008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698184.102031:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102038:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698184.102047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698184.102053:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102061:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698184.102066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102075:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102084:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.102091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.102095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698184.102099:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126306944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698184.102115:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102123:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698184.102134:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102140:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698184.102150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698184.102156:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102163:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698184.102167:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102183:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.102189:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.102192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698184.102195:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698184.102205:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698184.102255:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102263:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698184.102274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698184.102280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102288:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698184.102292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.102318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.102320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698184.102324:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698184.102337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102345:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698184.102356:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102363:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698184.102374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698184.102381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698209 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102388:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698184.102393:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698184/real 1787698184] req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698184.102408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698184.102414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698184.102416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698184.102443:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698184.102451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698184.102456:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126306944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698184.102464:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698184.102470:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126307072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698184.102482:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698184.102487:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126307200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698184.102492:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698184.102496:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126307328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698209 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698184.102502:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698184.613501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698184.613506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698184.613511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698184.741594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698185.637516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698185.637519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698185.637547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698185.765645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698186.021516:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300759407, at 4300759424 00000100:00000040:2.0:1787698186.021541:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300759407, at 4300759424 00000100:00100000:1.0:1787698186.661565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698186.661584:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698186.661596:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698186.789583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698187.685497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698187.685554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698187.685557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698187.813614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698188.709553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698188.709560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698188.709566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698188.837545:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698189.221521:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698189.221531:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698189.221535:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698189.221547:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6090 00000100:00080000:0.0:1787698189.221561:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698189.221565:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221570:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221575:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698189.221581:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698189.221584:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221587:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221616:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698189.221620:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221634:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698189.221671:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698189.221685:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698189.221705:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698189.221709:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698189.221711:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698189.221715:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6090 00000100:00080000:0.0:1787698189.221719:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698189.221722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221725:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221728:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698189.221731:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698189.221735:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221738:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221740:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698189.221743:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698189.221750:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698189.221755:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698189.221762:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698189.221774:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126307456:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698189.221775:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698189.221787:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698189.221806:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698189.221810:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698189.221812:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698189.221816:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6090 00000100:00000040:1.0:1787698189.221821:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698189.221823:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698189.221829:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221836:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698189.221840:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221841:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698189.221844:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698189.221847:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221850:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698189.221852:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698189.221856:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698189.221860:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698189.221871:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698189.221881:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698189.221890:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698189.221893:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698189.221895:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698189.221900:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6090 00000100:00080000:0.0:1787698189.221904:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698189.221911:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221914:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221916:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698189.221918:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698189.221921:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.221923:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.221925:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698189.221927:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698189.221932:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698189.221935:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698189.221945:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698189.221950:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698189.221957:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698189.221971:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698189.221973:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698189.221975:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698189.221979:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307584:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698189.221987:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6090 00000100:00080000:0.0:1787698189.221993:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698189.221996:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698189.222000:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.222004:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787698189.222004:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698189.222008:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698189.222016:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698189.222021:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698189.222023:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698189.222027:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698189.222034:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698189.222039:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698189.222039:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:0.0:1787698189.222047:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698189.222051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698189.222058:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698189.222068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698189.222078:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698189.222086:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698189.222088:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6100) 00000100:00000040:1.0:1787698189.222098:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698189.222113:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787698189.222123:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698189.222146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698189.222166:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698189.222178:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698189.222192:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787698189.222198:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698189.222207:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698189.222218:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698189.222226:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698189.222236:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698189.733558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698189.733563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698189.733567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698189.861621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698190.757553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698190.757553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698190.757559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698190.885608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698191.141568:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300764527, at 4300764544 00000100:00000040:3.0:1787698191.141657:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300764527, at 4300764544 00000100:00100000:3.0:1787698191.781538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698191.781541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698191.781556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698191.909604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698192.293722:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698192.293773:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698192.293807:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698192.293850:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698192.293872:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698192.293885:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.293897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698189/real 1787698192] req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698192.293900:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698192.293913:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698192.293927:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787698192.293937:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698192.293941:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698192.293968:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787698192.293969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698192.293980:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698214 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698192.293984:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698192.293995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698192.294002:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307584:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698192.294020:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698189/real 1787698192] req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294032:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698192.294040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698192.294054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698192.294057:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698192.294060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294068:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307712:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698192.294100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698189/real 1787698192] req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698192.294119:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698192.294126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698192.294129:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698192.294133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294141:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126307456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698192.294147:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126307584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698192.294159:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698192.294170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126307712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698192.294181:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698192.294185:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126307456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698192.294191:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698192.294199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698189/real 1787698192] req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698192.294216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698192.294223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698192.294227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698192.294230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307840:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698192.294248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698189/real 1787698192] req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698192.294267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698192.294279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698192.294282:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698192.294286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698192.294292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126307968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698192.294297:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126307840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698192.294304:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698192.294310:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126307968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698214 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698192.294317:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698192.805545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698192.805546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698192.805546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698192.933593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698193.829554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698193.829554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698193.829558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698193.957577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698194.341554:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698194.341565:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698194.341577:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698194.341591:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6095 00000100:00080000:2.0:1787698194.341603:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698194.341606:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341612:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698194.341617:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698194.341619:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341621:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341623:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698194.341626:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341639:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698194.341665:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698194.341679:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698194.341700:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698194.341703:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698194.341704:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698194.341708:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6095 00000100:00080000:2.0:1787698194.341710:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698194.341711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341713:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341714:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698194.341716:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698194.341718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341721:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698194.341722:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698194.341732:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698194.341736:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698194.341745:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698194.341755:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698194.341762:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698194.341765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698194.341767:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698194.341771:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6095 00000100:00080000:2.0:1787698194.341775:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698194.341778:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341780:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341783:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698194.341785:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698194.341788:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341790:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341793:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698194.341796:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698194.341801:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698194.341806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698194.341816:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698194.341826:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698194.341830:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.341857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126308096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698194.341862:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698194.341867:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698194.341869:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698194.341875:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6095 00000100:00080000:2.0:1787698194.341878:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698194.341881:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341885:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698194.341888:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698194.341890:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341893:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341894:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698194.341900:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698194.341901:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698194.341907:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698194.341916:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698194.341917:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698194.341932:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698194.341941:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698194.341949:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698194.341951:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698194.341953:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698194.341957:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6095 00000100:00080000:2.0:1787698194.341962:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698194.341966:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341968:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341970:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698194.341972:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:1.0:1787698194.341977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341980:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698194.341985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698194.341987:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698194.341990:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698194.341996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698194.342003:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698194.342003:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698194.342017:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698194.342031:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698194.342057:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6105) 00000100:00100000:1.0:1787698194.342072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698194/real 1787698194] req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342093:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698194.342113:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698194.342117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698194.342121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126308096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698194.342139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342148:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698194.342177:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342184:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698194.342197:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698194.342204:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342214:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698194.342220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698194/real 1787698194] req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342231:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698194.342245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698194.342248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698194.342251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698194.342264:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126308096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698194.342274:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698194.342286:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126308224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698194.342292:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698194.342307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698194.342329:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342337:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698194.342348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698194.342355:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342364:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698194.342370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698194/real 1787698194] req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342380:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698194.342396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698194.342400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698194.342404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698194.342417:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698194.342479:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342485:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698194.342496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698194.342503:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342516:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698194.342521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698194/real 1787698194] req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342530:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342538:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698194.342544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698194.342547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698194.342550:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698194.342561:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698194.342581:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342588:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698194.342598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698194.342605:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698219 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342614:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698194.342621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698194/real 1787698194] req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698194.342639:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698194.342646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698194.342649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698194.342653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698194.342660:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698194.342667:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126308352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698194.342675:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698194.342680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126308480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698194.342687:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698194.342699:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126308608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698219 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698194.342707:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698194.853550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698194.853553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698194.853582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698194.981580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698195.877542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698195.877542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698195.877542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698196.005580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698196.263283:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300769647, at 4300769665 00000100:00000040:2.0:1787698196.263644:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300769647, at 4300769666 00000100:00100000:1.0:1787698196.901532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698196.901532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698196.902201:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698197.029611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698197.925517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698197.925517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698197.925519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698198.053580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698198.949597:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698198.949597:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698198.950075:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698199.077567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698199.461568:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698199.461583:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698199.461588:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698199.461601:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6100 00000100:00080000:0.0:1787698199.461621:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698199.461625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.461629:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461633:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698199.461640:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698199.461643:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.461645:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461655:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698199.461658:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461677:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698199.461709:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698199.461753:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698199.461785:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698199.461789:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698199.461790:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698199.461794:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6100 00000100:00080000:0.0:1787698199.461797:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698199.461800:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.461802:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461805:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698199.461807:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698199.461809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.461811:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461813:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698199.461815:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698199.461823:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698199.461825:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698199.461831:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698199.461842:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698199.461846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126308736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698199.461854:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698199.461861:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698199.461864:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698199.461865:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698199.461873:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6100 00000100:00080000:0.0:1787698199.461876:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698199.461878:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.461880:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461881:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698199.461884:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698199.461888:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698199.461888:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698199.461894:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461898:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698199.461902:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698199.461908:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787698199.461909:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698199.461914:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698199.461922:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698199.461928:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698199.461934:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698199.461936:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698199.461937:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698199.461940:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6100 00000100:00080000:0.0:1787698199.461943:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698199.461944:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.461946:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461948:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000800:00000100:1.0:1787698199.461967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461971:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698199.461976:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.461982:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.461984:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698199.461986:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698199.461991:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698199.461992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698199.461995:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698199.462005:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698199.462012:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698199.462018:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698199.462020:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698199.462021:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698199.462024:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6100 00000100:00080000:0.0:1787698199.462028:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698199.462032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698199/real 1787698199] req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698199.462033:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.462037:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.462042:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698199.462045:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698199.462048:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698199.462050:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698199.462052:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698199.462053:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698199.462057:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698199.462058:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698199.462063:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698199.462071:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698199.462073:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698199.462079:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698199.462086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698199.462090:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462110:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126308736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698199.462111:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6110) 00000100:00000040:1.0:1787698199.462130:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126308736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698199.462138:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698199.462153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698199.462169:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462174:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698199.462183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698199.462187:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462194:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698199.462197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698199/real 1787698199] req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698199.462218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698199.462222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698199.462226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698199.462245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698199.462266:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462272:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698199.462281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698199.462285:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462291:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698199.462294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698199/real 1787698199] req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698199.462311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698199.462313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698199.462315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126308992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698199.462324:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698199.462337:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462343:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698199.462352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698199.462358:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462368:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698199.462373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698199/real 1787698199] req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698199.462398:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698199.462400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698199.462402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698199.462410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698199.462456:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462463:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698199.462475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698199.462482:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698224 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462490:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698199.462496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698199/real 1787698199] req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698199.462511:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698199.462515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698199.462516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698199.462518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698199.462522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698199.462526:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126308864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698199.462531:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698199.462535:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126308992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698199.462540:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698199.462543:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126309120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698199.462548:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698199.462550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126309248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698224 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698199.462554:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698199.973476:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698199.973479:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698199.973510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698200.101583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698200.997553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698200.997554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698200.997554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698201.125578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698201.381523:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300774767, at 4300774784 00000100:00000040:3.0:1787698201.381523:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300774767, at 4300774784 00000100:00100000:1.0:1787698202.021521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698202.021522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698202.021543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698202.149583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698203.045539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698203.045541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698203.045545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698203.173600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698204.069534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698204.069534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698204.069537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698204.197540:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698204.581526:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698204.581536:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698204.581538:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698204.581546:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6105 00000100:00080000:2.0:1787698204.581555:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698204.581557:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581562:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698204.581566:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698204.581568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581570:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581572:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698204.581574:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581584:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698204.581603:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698204.581613:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698204.581628:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698204.581630:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698204.581632:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698204.581634:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6105 00000100:00080000:2.0:1787698204.581636:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698204.581638:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581641:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581642:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698204.581644:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698204.581646:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581648:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581649:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698204.581651:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698204.581656:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.581670:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126309376:192.168.201.136@tcp:250:kworker.0 00000008:00100000:2.0:1787698204.581671:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698204.581677:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698204.581685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698204.581691:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698204.581697:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698204.581698:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698204.581700:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698204.581702:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698204.581707:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6105 00000100:00080000:2.0:1787698204.581709:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698204.581711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581712:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581714:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698204.581715:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698204.581720:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581723:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581726:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698204.581729:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698204.581733:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:2.0:1787698204.581733:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698204.581736:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698204.581745:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698204.581750:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698204.581762:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698204.581765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698204.581766:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698204.581771:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6105 00000800:00000100:1.0:1787698204.581777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698204.581779:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698204.581784:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581788:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581790:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698204.581793:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698204.581799:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698204.581800:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698204.581801:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581805:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698204.581809:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698204.581816:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698204.581820:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698204.581828:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698204.581835:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698204.581840:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698204.581844:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698204/real 1787698204] req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698204.581844:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698204.581847:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698204.581852:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6105 00000100:00080000:2.0:1787698204.581856:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698204.581860:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581876:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581878:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698204.581879:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698204.581881:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698204.581884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698204.581885:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698204.581888:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698204.581892:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698204.581893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698204.581898:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698204.581907:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698204.581911:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698204.581920:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698204.581925:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698204.581927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698204.581934:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698204.581937:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6115) 00000100:00100000:1.0:1787698204.581944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126309376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698204.581949:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126309376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698204.581956:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698204.581968:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.581976:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698204.581996:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582004:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698204.582016:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698204.582021:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582032:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698204.582035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698204/real 1787698204] req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582047:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698204.582052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698204.582053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698204.582055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698204.582063:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698204.582075:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582079:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698204.582085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698204.582088:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582093:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698204.582096:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698204/real 1787698204] req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582107:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698204.582112:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698204.582113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698204.582115:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698204.582122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582126:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698204.582132:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582136:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698204.582140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698204.582144:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582149:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698204.582151:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698204/real 1787698204] req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698204.582170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698204.582172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698204.582174:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698204.582181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698204.582191:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582195:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698204.582199:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698204.582202:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698229 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582207:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698204.582209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698204/real 1787698204] req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698204.582220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698204.582225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698204.582226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698204.582228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698204.582232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126309888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698204.582235:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126309504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698204.582239:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698204.582242:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126309632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698204.582246:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698204.582248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126309760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698204.582252:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698204.582254:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126309888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698229 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698204.582258:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698205.093509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698205.093509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698205.093509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698205.221567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698206.117567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698206.117567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698206.117578:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698206.245625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698206.501522:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300779887, at 4300779904 00000100:00000040:3.0:1787698206.501533:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300779887, at 4300779904 00000100:00100000:0.0:1787698207.141522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698207.141522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698207.141522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698207.269600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698208.165519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698208.165519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698208.165522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698208.293550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698209.189521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698209.189521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698209.189546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698209.317632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698209.701552:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698209.701563:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698209.701568:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698209.701576:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6110 00000100:00080000:3.0:1787698209.701586:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698209.701588:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701592:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701595:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698209.701598:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698209.701601:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701604:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701606:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698209.701609:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701622:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698209.701642:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698209.701653:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698209.701686:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698209.701690:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698209.701692:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698209.701697:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6110 00000100:00080000:3.0:1787698209.701702:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698209.701708:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701711:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701715:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698209.701717:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698209.701719:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701722:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701724:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698209.701726:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698209.701735:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698209.701738:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698209.701740:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698209.701756:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698209.701771:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.701785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126310016:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698209.701805:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698209.701808:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698209.701809:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698209.701813:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6110 00000100:00080000:3.0:1787698209.701816:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698209.701825:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701827:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698209.701835:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698209.701835:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698209.701841:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698209.701850:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701863:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701868:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698209.701875:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698209.701885:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698209.701885:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698209.701894:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698209.701907:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698209.701918:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698209.701926:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698209.701928:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698209.701929:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698209.701933:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6110 00000100:00080000:3.0:1787698209.701940:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698209.701942:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701943:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701945:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698209.701946:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:1.0:1787698209.701954:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701955:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.701964:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.701966:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698209.701973:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698209.701978:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698209.701981:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698209.701988:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698209.701990:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698209.701999:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698209.702025:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698209.702036:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698209.702037:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698209.702043:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6110 00000100:00080000:3.0:1787698209.702048:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698209.702051:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698209.702056:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.702059:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:1.0:1787698209.702066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698209/real 1787698209] req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698209.702071:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698209.702079:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698209.702086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698209.702086:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698209.702093:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698209.702105:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698209.702113:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698209.702115:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698209.702119:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698209.702134:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698209.702136:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698209.702150:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698209.702151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698209.702161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698209.702166:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6120) 00000100:00100000:1.0:1787698209.702177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126310016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698209.702188:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126310016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698209.702199:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698209.702219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698209.702239:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702244:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698209.702254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698209.702258:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702266:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698209.702270:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698209/real 1787698209] req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698209.702287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698209.702289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698209.702291:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698209.702304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702309:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698209.702317:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702323:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698209.702332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698209.702339:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702347:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698209.702353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698209/real 1787698209] req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702373:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698209.702379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698209.702381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698209.702383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698209.702393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698209.702408:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702413:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698209.702452:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698209.702456:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702463:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698209.702466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698209/real 1787698209] req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702478:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698209.702482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698209.702484:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698209.702486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698209.702493:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702499:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698209.702506:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702511:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698209.702516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698209.702520:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698234 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702525:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698209.702528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698209/real 1787698209] req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698209.702541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698209.702549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698209.702552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698209.702556:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698209.702563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698209.702569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126310144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698209.702591:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698209.702599:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126310272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698209.702606:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698209.702611:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126310400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698209.702619:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698209.702624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126310528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698234 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698209.702630:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698210.213560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698210.213593:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698210.213595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698210.341622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698211.237542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698211.237542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698211.237642:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698211.365614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698211.621508:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300785007, at 4300785024 00000100:00000040:2.0:1787698211.621510:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300785007, at 4300785024 00000100:00100000:1.0:1787698212.261537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698212.261537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698212.261557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698212.389599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698213.285517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698213.285518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698213.285521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698213.413591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698214.309536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698214.309536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698214.309536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698214.437593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698214.821537:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698214.821546:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698214.821551:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698214.821560:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6115 00000100:00080000:2.0:1787698214.821572:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698214.821574:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821578:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821582:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698214.821586:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698214.821588:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821590:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821592:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698214.821594:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821607:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698214.821633:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698214.821645:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698214.821661:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698214.821663:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698214.821664:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698214.821667:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6115 00000100:00080000:2.0:1787698214.821669:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698214.821671:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821673:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821674:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698214.821676:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698214.821678:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821679:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821680:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698214.821682:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698214.821702:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698214.821707:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698214.821716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698214.821717:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698214.821747:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698214.821754:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698214.821756:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126310656:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698214.821757:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698214.821761:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698214.821766:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6115 00000100:00080000:2.0:1787698214.821768:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698214.821770:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821772:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821773:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698214.821775:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698214.821777:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821779:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821781:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698214.821782:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698214.821785:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698214.821788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698214.821795:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698214.821801:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698214.821803:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698214.821829:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698214.821830:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698214.821834:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698214.821836:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698214.821840:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6115 00000100:00080000:2.0:1787698214.821842:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698214.821843:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821845:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821846:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698214.821847:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698214.821849:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821851:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821852:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698214.821854:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698214.821857:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698214.821861:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698214.821865:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698214.821871:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698214.821875:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698214.821882:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698214.821885:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698214.821886:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698214.821889:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6115 00000100:00080000:2.0:1787698214.821892:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698214.821895:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698214.821896:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698214.821899:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821903:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698214.821907:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698214.821912:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698214.821915:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698214.821916:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698214.821918:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698214.821922:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698214.821925:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698214.821932:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698214.821938:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698214/real 1787698214] req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698214.821939:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698214.821964:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6125) 00000100:00100000:1.0:1787698214.821969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.821978:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698214.821985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698214.821988:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698214.821990:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.821996:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126310656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698214.822001:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126310656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698214.822008:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698214.822021:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698214.822036:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822041:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698214.822048:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698214.822052:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822063:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698214.822066:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698214/real 1787698214] req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822078:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698214.822082:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698214.822084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698214.822086:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698214.822094:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698214.822105:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822109:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698214.822113:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698214.822116:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822122:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698214.822125:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698214/real 1787698214] req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698214.822141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698214.822143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698214.822144:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126310912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698214.822152:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822160:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698214.822171:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822177:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698214.822188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698214.822194:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822203:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698214.822209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698214/real 1787698214] req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698214.822244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698214.822247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698214.822252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698214.822265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698214.822287:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822294:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698214.822306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698214.822313:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698239 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822320:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698214.822324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698214/real 1787698214] req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822331:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698214.822337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698214.822341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698214.822343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698214.822352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698214.822356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698214.822360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126310784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698214.822366:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698214.822371:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126310912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698214.822376:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698214.822380:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126311040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698214.822383:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698214.822386:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126311168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698239 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698214.822389:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698215.333518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698215.333518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698215.333519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698215.461614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698216.357536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698216.357536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698216.357578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698216.485601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698216.741537:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300790127, at 4300790144 00000100:00000040:3.0:1787698216.741537:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300790127, at 4300790144 00000100:00100000:3.0:1787698217.381481:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698217.381489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698217.381505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698217.509703:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698218.405517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698218.405519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698218.405531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698218.533511:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698219.429556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698219.429556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698219.429556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698219.557662:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698219.941581:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698219.941595:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698219.941613:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698219.941627:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6120 00000100:00080000:2.0:1787698219.941642:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698219.941647:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698219.941652:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.941657:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698219.941662:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698219.941666:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698219.941668:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.941673:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698219.941676:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698219.941696:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698219.941731:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698219.941749:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698219.941778:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698219.941782:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698219.941785:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698219.941789:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6120 00000100:00080000:2.0:1787698219.941795:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698219.941800:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698219.941804:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.941807:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698219.941810:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698219.941815:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698219.941818:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.941821:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787698219.941832:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698219.941832:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698219.941850:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698219.941858:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698219.941859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126311296:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698219.941878:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698219.941893:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.941901:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.941956:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698219.941959:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698219.941964:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698219.941966:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698219.941973:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6120 00000100:00080000:2.0:1787698219.941978:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698219.941982:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698219.941985:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.941987:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698219.941990:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698219.941993:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698219.941999:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698219.942003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698219.942005:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698219.942011:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698219.942017:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698219.942023:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698219.942026:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698219.942039:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698219.942061:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942063:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698219/real 1787698219] req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698219.942115:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698219.942117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698219.942123:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698219.942128:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698219.942143:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6120 00000100:00080000:2.0:1787698219.942149:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698219.942151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698219.942164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698219.942168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698219.942171:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698219.942179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698219.942185:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.942195:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:1.0:1787698219.942196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126311296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698219.942202:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126311296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698219.942210:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698219.942226:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698219.942230:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698219.942236:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.942241:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698219.942246:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698219.942249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698219.942260:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698219.942262:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311424:192.168.201.136@tcp:38:kworker.0 00000020:00000040:2.0:1787698219.942277:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698219.942282:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942290:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698219.942295:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698219.942306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698219.942315:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942325:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:2.0:1787698219.942334:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698219/real 1787698219] req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698219.942388:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698219.942392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698219.942403:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698219.942407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698219.942408:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698219.942411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698219.942445:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698219.942458:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6120 00000100:00100000:1.0:1787698219.942465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311424:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698219.942467:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698219.942476:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698219.942483:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698219.942488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698219.942489:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698219.942494:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698219.942504:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698219.942513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698219.942514:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698219.942520:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698219.942529:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698219.942534:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698219.942538:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698219.942543:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698219.942553:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:1.0:1787698219.942568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698219.942572:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698219.942576:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942615:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698219.942621:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698219/real 1787698219] req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698219.942640:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942678:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698219.942694:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698219.942696:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6131) 00000100:00080000:1.0:1787698219.942697:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698219.942704:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698219.942714:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126311424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698219.942721:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698219.942726:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126311552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698219.942731:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698219.942737:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698219.942751:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942756:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698219.942763:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698219.942767:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942772:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698219.942775:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698219/real 1787698219] req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698219.942792:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698219.942795:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698219.942799:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942807:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698219.942813:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942821:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698219.942833:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942840:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698219.942848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698219.942852:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698244 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942858:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698219.942861:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698219/real 1787698219] req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698219.942872:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698219.942876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698219.942878:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698219.942880:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698219.942884:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126311808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698219.942887:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126311680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698219.942892:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698219.942895:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126311808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698244 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698219.942899:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698220.453567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698220.453567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698220.453639:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698220.581564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698221.477557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698221.477658:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698221.477692:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698221.605612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698221.861530:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300795247, at 4300795264 00000100:00000040:2.0:1787698221.861538:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300795247, at 4300795264 00000100:00100000:1.0:1787698222.501521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698222.501632:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698222.501711:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698222.629585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698223.525536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698223.525578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698223.525584:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698223.653581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698224.549498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698224.549500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698224.549534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698224.677607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698225.062627:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698225.062636:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698225.062641:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698225.062650:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6126 00000100:00080000:1.0:1787698225.062660:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698225.062664:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062668:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062671:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698225.062675:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698225.062678:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062680:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062683:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698225.062687:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062699:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698225.062723:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698225.062737:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.062760:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698225.062763:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698225.062765:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698225.062769:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6126 00000100:00080000:1.0:1787698225.062772:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698225.062775:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062778:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062780:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698225.062782:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698225.062785:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062794:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062796:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698225.062801:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698225.062810:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698225.062815:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698225.062825:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698225.062837:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698225.062838:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.062864:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126311936:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698225.062878:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698225.062881:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698225.062883:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698225.062887:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6126 00000100:00080000:1.0:1787698225.062891:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698225.062895:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698225.062896:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062899:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062901:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698225.062903:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698225.062905:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062909:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062913:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698225.062918:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698225.062925:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698225.062930:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062932:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698225.062942:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698225.062951:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.062958:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698225.062960:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698225.062962:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698225.062967:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6126 00000800:00000100:0.0:1787698225.062969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698225.062971:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698225.062975:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062978:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.062980:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698225.062982:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698225.062985:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.062991:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698225.062992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698225.062993:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698225.062999:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698225.063006:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698225.063010:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698225.063017:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698225.063025:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.063032:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698225.063035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698225.063036:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698225.063039:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698225.063047:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6126 00000100:00080000:1.0:1787698225.063051:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698225.063055:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.063058:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.063060:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698225.063062:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:0.0:1787698225.063065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698225.063067:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698225.063073:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698225.063077:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698225.063082:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698225.063086:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787698225.063088:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698225.063091:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698225.063100:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787698225.063102:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698225.063111:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698225.063115:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698225.063130:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6136) 00000100:00100000:0.0:1787698225.063136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126311936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698225.063144:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126311936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698225.063155:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698225.063170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063179:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698225.063193:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063211:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698225.063226:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698225.063233:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063244:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698225.063250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698225.063277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698225.063280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698225.063284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698225.063297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698225.063317:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063325:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698225.063343:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698225.063351:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063361:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698225.063367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063379:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063387:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698225.063395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698225.063398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698225.063402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698225.063416:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698225.063499:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063507:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698225.063531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698225.063539:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063560:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698225.063566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698225.063593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698225.063596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698225.063600:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698225.063613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063621:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698225.063632:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063639:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698225.063651:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698225.063658:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698250 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063666:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698225.063678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698225/real 1787698225] req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063690:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698225.063698:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698225.063704:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698225.063707:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698225.063711:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698225.063719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698225.063725:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126312064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698225.063732:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698225.063738:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126312192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698225.063745:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698225.063751:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126312320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698225.063770:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698225.063776:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126312448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698250 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698225.063783:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698225.573588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698225.573592:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698225.573593:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698225.701612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698226.597599:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698226.597599:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698226.597599:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698226.725594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698226.981536:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300800368, at 4300800384 00000100:00000040:3.0:1787698226.981536:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300800368, at 4300800384 00000100:00100000:3.0:1787698227.621489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698227.621515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698227.621539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698227.749609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698228.645571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698228.645573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698228.645615:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698228.774006:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698229.669538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698229.669538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698229.669547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698229.797648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698230.181649:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698230.181656:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698230.181659:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698230.181667:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6131 00000100:00080000:1.0:1787698230.181675:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698230.181677:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181679:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181682:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698230.181687:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698230.181694:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181697:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181700:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698230.181703:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181716:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698230.181742:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698230.181765:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698230.181790:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698230.181792:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698230.181793:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698230.181801:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6131 00000100:00080000:1.0:1787698230.181804:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698230.181806:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181808:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181809:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698230.181811:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698230.181813:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181814:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181816:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698230.181818:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698230.181826:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698230.181829:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698230.181837:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698230.181847:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698230.181847:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698230.181860:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698230.181866:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698230.181867:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698230.181877:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6131 00000100:00080000:1.0:1787698230.181882:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698230.181887:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126312576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698230.181887:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181894:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181898:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698230.181900:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698230.181903:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181906:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181908:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698230.181911:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698230.181916:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698230.181920:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698230.181922:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698230.181933:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698230.181935:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698230.181946:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698230.181954:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698230.181958:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698230.181960:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698230.181964:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6131 00000800:00000100:0.0:1787698230.181970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698230.181972:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698230.181977:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181979:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181981:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698230.181983:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698230.181987:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.181990:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.181991:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698230.181993:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698230.181996:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698230.182000:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698230.182001:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698230.182008:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698230.182021:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698230.182030:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698230.182033:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698230.182035:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787698230.182040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698230/real 1787698230] req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698230.182041:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6131 00000100:00080000:1.0:1787698230.182047:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698230.182054:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.182058:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.182061:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698230.182064:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:0.0:1787698230.182065:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698230.182070:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698230.182077:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698230.182083:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787698230.182088:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698230.182089:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698230.182097:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698230.182101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787698230.182102:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698230.182109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:1.0:1787698230.182113:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698230.182118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698230.182124:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182144:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126312576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698230.182144:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6141) 00000100:00000040:0.0:1787698230.182153:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126312576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698230.182175:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698230.182189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698230.182211:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182218:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698230.182229:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698230.182235:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182245:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698230.182250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698230/real 1787698230] req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182260:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698230.182274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698230.182277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698230.182281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182287:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698230.182302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698230.182320:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182326:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698230.182336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698230.182342:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182350:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698230.182355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698230/real 1787698230] req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698230.182379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698230.182382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698230.182385:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698230.182396:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698230.182461:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182468:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698230.182481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698230.182487:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182496:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698230.182501:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698230/real 1787698230] req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698230.182526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698230.182529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698230.182532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126312960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698230.182544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698230.182567:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182573:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698230.182583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698230.182589:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698255 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182597:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698230.182602:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698230/real 1787698230] req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182611:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698230.182619:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698230.182625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698230.182628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698230.182632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698230.182638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698230.182644:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126312704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698230.182651:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698230.182656:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126312832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698230.182672:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698230.182677:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126312960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698230.182684:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698230.182688:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126313088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698255 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698230.182694:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698230.693559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698230.693562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698230.693730:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698230.821621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698231.717526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698231.717527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698231.717552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698231.845594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698232.101517:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300805487, at 4300805504 00000100:00000040:3.0:1787698232.101517:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300805487, at 4300805504 00000100:00100000:1.0:1787698232.741560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698232.741573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698232.741575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698232.869704:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698233.765520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698233.765520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698233.765546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698233.893587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698234.789526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698234.789530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698234.789545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698234.917589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698235.301533:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698235.301543:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698235.301546:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698235.301555:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6136 00000100:00080000:1.0:1787698235.303387:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698235.303392:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303397:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303400:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698235.303404:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698235.303407:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303411:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303413:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698235.303416:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303440:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698235.303463:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698235.303486:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698235.303513:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698235.303516:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698235.303518:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698235.303522:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6136 00000100:00080000:1.0:1787698235.303526:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698235.303528:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303531:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303533:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698235.303536:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698235.303539:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303541:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303543:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698235.303546:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698235.303556:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698235.303561:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698235.303570:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698235.303573:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698235.303587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126313216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698235.303589:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698235.303598:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698235.303600:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698235.303602:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698235.303605:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6136 00000100:00080000:1.0:1787698235.303608:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698235.303613:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698235.303613:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303616:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303620:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698235.303623:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698235.303628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698235.303631:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303634:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303636:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698235.303638:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698235.303642:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698235.303645:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698235.303651:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698235.303657:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698235.303664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698235.303666:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698235.303669:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698235.303670:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698235.303673:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6136 00000100:00080000:1.0:1787698235.303676:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698235.303678:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303680:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303681:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698235.303683:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698235.303686:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698235.303687:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303692:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303696:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698235.303700:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698235.303704:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698235.303708:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698235.303714:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698235.303721:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.303726:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698235/real 1787698235] req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698235.303729:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698235.303734:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698235.303736:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698235.303741:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6136 00000100:00080000:1.0:1787698235.303743:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698235.303748:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698235.303749:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698235.303751:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303755:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698235.303758:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698235.303760:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698235.303762:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698235.303763:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698235.303765:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698235.303768:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698235.303773:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698235.303781:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698235.303785:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698235.303796:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698235.303798:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698235.303806:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698235.303808:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6146) 00000100:00000040:0.0:1787698235.303814:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.303823:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126313216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698235.303830:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126313216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698235.303840:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698235.303854:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.303862:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698235.303873:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.303878:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698235.303886:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698235.303892:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.303902:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698235.303908:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698235/real 1787698235] req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.303933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.303942:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698235.303949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698235.303952:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698235.303956:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.303964:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698235.303970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.303978:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698235.303990:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.303998:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698235.304010:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698235.304017:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304026:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698235.304029:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698235/real 1787698235] req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304050:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698235.304054:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698235.304056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698235.304058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698235.304066:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698235.304079:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304083:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698235.304090:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698235.304094:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304099:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698235.304102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698235/real 1787698235] req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698235.304126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698235.304128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698235.304130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698235.304136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698235.304147:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304152:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698235.304158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698235.304161:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698260 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304167:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698235.304171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698235/real 1787698235] req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304181:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698235.304195:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698235.304202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698235.304205:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698235.304209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698235.304216:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698235.304222:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126313344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698235.304230:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698235.304237:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126313472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698235.304243:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698235.304249:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126313600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698235.304256:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698235.304261:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126313728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698260 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698235.304268:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698235.813525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698235.813525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698235.813528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698235.941639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698236.837479:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698236.837504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698236.837519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698236.965569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698237.221495:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300810609, at 4300810624 00000100:00000040:2.0:1787698237.221501:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300810609, at 4300810624 00000100:00100000:0.0:1787698237.861532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698237.861532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698237.861532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698237.989786:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698238.885585:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698238.891156:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698238.891631:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698239.013587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698239.909514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698239.909514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698239.909514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698240.037542:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698240.421541:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698240.421551:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698240.421555:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698240.421564:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6141 00000100:00080000:3.0:1787698240.421575:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698240.421578:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.421582:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421586:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698240.421590:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698240.421593:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.421595:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421598:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698240.421601:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421615:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698240.421641:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698240.421657:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698240.421678:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698240.421682:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698240.421700:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698240.421705:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6141 00000100:00080000:3.0:1787698240.421710:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698240.421713:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.421716:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421719:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698240.421722:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698240.421726:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698240.421727:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.421733:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421738:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698240.421745:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698240.421750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126313856:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698240.421755:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698240.421762:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698240.421775:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698240.421781:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.421799:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698240.421803:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698240.421814:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698240.421818:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698240.421819:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698240.421823:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6141 00000100:00080000:3.0:1787698240.421828:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698240.421831:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.421834:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:0.0:1787698240.421838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421839:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698240.421844:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698240.421854:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698240.421858:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698240.421860:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421865:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698240.421871:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698240.421878:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698240.421883:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698240.421894:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698240.421901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698240/real 1787698240] req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698240.421909:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698240.421921:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698240.421926:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698240.421926:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698240.421930:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698240.421939:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6141 00000100:00000040:0.0:1787698240.421945:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698240.421945:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698240.421953:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.421959:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698240.421961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787698240.421962:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698240.421967:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:0.0:1787698240.421969:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698240.421973:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698240.421979:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698240.421982:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.421987:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698240.421993:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698240.421999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126313856:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698240.422001:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698240.422008:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698240.422010:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126313856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698240.422020:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787698240.422023:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698240.422038:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422042:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698240.422048:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698240.422056:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698240.422061:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698240.422068:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6141 00000100:00100000:0.0:1787698240.422071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313984:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698240.422074:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698240.422080:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.422083:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.422086:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698240.422088:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698240.422092:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698240.422095:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698240.422098:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698240.422100:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698240.422103:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698240.422111:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787698240.422117:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698240.422119:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698240.422132:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698240.422137:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698240.422150:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422151:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698240.422181:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6151) 00000100:00000040:0.0:1787698240.422184:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698240.422192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698240/real 1787698240] req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698240.422223:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698240.422226:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698240.422230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126313984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698240.422243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698240.422301:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422309:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698240.422321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698240.422328:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422338:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698240.422344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698240/real 1787698240] req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698240.422371:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698240.422374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698240.422377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698240.422390:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126313984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698240.422407:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698240.422414:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126314112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698240.422450:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698240.422459:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698240.422480:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422487:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698240.422499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698240.422506:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422515:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698240.422521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698240/real 1787698240] req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698240.422549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698240.422551:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698240.422555:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698240.422574:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422583:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698240.422593:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422601:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698240.422610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698240.422616:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698265 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422624:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698240.422629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698240/real 1787698240] req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698240.422641:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698240.422645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698240.422647:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698240.422649:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698240.422652:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698240.422655:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126314240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698240.422668:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698240.422673:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126314368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698265 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698240.422679:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698240.933521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698240.933522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698240.933652:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698241.061584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698241.957502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698241.957507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698241.957531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698242.085575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698242.341509:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300815727, at 4300815744 00000100:00000040:2.0:1787698242.341514:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300815727, at 4300815744 00000100:00100000:1.0:1787698242.981507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698242.981511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698242.981512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698243.109586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698244.005495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698244.005495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698244.005505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698244.133553:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698245.029525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698245.029525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698245.029540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698245.157568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698245.541502:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698245.541512:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698245.541516:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698245.541527:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6146 00000100:00080000:2.0:1787698245.541539:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698245.541543:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541548:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541552:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698245.541557:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698245.541561:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541564:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541567:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698245.541571:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541589:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698245.541618:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698245.541631:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698245.541649:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698245.541652:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698245.541653:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698245.541656:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6146 00000100:00080000:2.0:1787698245.541659:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698245.541661:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541662:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541664:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698245.541666:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698245.541668:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541670:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541671:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698245.541673:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698245.541682:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698245.541686:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698245.541695:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698245.541702:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698245.541708:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698245.541710:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698245.541711:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698245.541714:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6146 00000100:00080000:2.0:1787698245.541716:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698245.541719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698245.541720:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541725:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541728:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698245.541731:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698245.541737:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541739:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541741:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00100000:0.0:1787698245.541743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126314496:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698245.541744:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698245.541751:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698245.541754:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698245.541760:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698245.541766:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698245.541772:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698245.541775:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:0.0:1787698245.541778:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698245.541779:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698245.541785:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6146 00000100:00080000:2.0:1787698245.541790:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698245.541794:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698245.541798:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541800:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541802:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698245.541804:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698245.541806:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541807:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541809:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698245.541810:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698245.541813:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698245.541817:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698245.541822:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698245.541828:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698245.541832:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698245.541834:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698245.541835:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698245.541837:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6146 00000100:00080000:2.0:1787698245.541841:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698245.541842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541848:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541850:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698245.541852:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698245.541854:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698245.541856:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698245.541857:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698245.541859:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698245.541862:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698245.541865:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698245.541869:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698245.541872:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698245.541881:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698245.541899:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6156) 00000100:00100000:0.0:1787698245.541922:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698245/real 1787698245] req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.541936:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.541942:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698245.541948:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698245.541951:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698245.541954:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.541981:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126314496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698245.541985:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.541991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698245.542002:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542011:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698245.542019:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698245.542024:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542031:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698245.542035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698245/real 1787698245] req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542042:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542048:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698245.542053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698245.542054:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698245.542056:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542060:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698245.542068:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126314496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698245.542076:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698245.542083:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126314624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698245.542088:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698245.542098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698245.542111:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542116:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698245.542122:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698245.542126:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542132:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698245.542135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698245/real 1787698245] req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542142:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698245.542152:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698245.542158:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698245.542160:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698245.542184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698245.542200:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542205:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698245.542212:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698245.542217:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698245.542226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698245/real 1787698245] req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698245.542242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698245.542244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698245.542246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126314880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698245.542259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698245.542271:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542275:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698245.542281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698245.542285:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698270 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542290:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698245.542293:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698245/real 1787698245] req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542300:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698245.542305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698245.542309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698245.542311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698245.542313:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698245.542317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698245.542323:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126314752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698245.542328:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698245.542332:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126314880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698245.542337:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698245.542340:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126315008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698270 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698245.542344:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698246.053487:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698246.053488:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698246.053491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698246.181657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698247.077531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698247.077536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698247.077549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698247.205562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698247.461543:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300820847, at 4300820864 00000100:00000040:3.0:1787698247.461543:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300820847, at 4300820864 00000100:00100000:0.0:1787698248.101569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698248.101573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698248.101576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698248.229627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698249.125523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698249.125523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698249.125540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698249.253602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698250.149532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698250.149532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698250.149533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698250.277636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698250.661507:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698250.661517:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698250.661522:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698250.661533:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6151 00000100:00080000:2.0:1787698250.661553:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698250.661557:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661563:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661568:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698250.661575:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698250.661579:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661582:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661585:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698250.661589:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661605:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698250.661633:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698250.661647:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698250.661665:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698250.661668:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698250.661670:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698250.661673:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6151 00000100:00080000:2.0:1787698250.661676:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698250.661678:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661680:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661682:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698250.661684:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698250.661686:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661689:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661690:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698250.661693:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698250.661702:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698250.661706:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698250.661706:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698250.661717:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698250.661723:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126315136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698250.661731:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698250.661740:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698250.661744:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698250.661746:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698250.661750:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6151 00000100:00080000:2.0:1787698250.661754:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698250.661758:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698250.661762:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698250.661764:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661770:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698250.661774:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698250.661780:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698250.661785:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661785:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661789:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698250.661791:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698250.661795:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698250.661798:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698250.661806:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698250.661813:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698250.661819:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698250.661821:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698250.661823:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698250.661826:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6151 00000800:00000100:0.0:1787698250.661831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698250.661832:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698250.661836:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661838:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661840:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698250.661842:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698250.661845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661846:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661848:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698250.661850:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698250.661854:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698250.661856:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698250.661862:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698250.661870:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698250.661875:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698250.661881:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698250.661884:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698250.661885:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698250.661889:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6151 00000100:00100000:0.0:1787698250.661890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698250/real 1787698250] req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698250.661892:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698250.661898:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661902:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661904:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698250.661905:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698250.661908:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698250.661909:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698250.661910:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698250.661911:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698250.661914:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698250.661920:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698250.661929:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698250.661940:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.661941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698250.661956:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6161) 00000100:00000040:0.0:1787698250.661963:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698250.661974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698250.661979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698250.661982:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.661987:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126315136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698250.661992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126315136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698250.662000:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698250.662014:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698250.662031:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662035:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698250.662044:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698250.662049:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662064:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698250.662068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698250/real 1787698250] req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662102:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698250.662107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698250.662109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698250.662111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698250.662119:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662124:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698250.662132:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662136:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698250.662146:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698250.662152:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662161:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698250.662166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698250/real 1787698250] req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662191:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662199:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698250.662207:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698250.662210:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698250.662214:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698250.662227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698250.662249:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662254:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698250.662263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698250.662268:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662274:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698250.662277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698250/real 1787698250] req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698250.662303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698250.662304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698250.662306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698250.662315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662319:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698250.662326:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662330:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698250.662336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698250.662341:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698275 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662350:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698250.662355:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698250/real 1787698250] req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698250.662379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698250.662387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698250.662389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698250.662391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698250.662396:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698250.662399:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126315264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698250.662404:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698250.662410:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126315392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698250.662417:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698250.662450:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126315520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698250.662455:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698250.662458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126315648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698275 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698250.662462:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698251.173531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698251.173531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698251.173531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698251.301632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698252.197552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698252.197552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698252.197567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698252.325607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698252.581546:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300825967, at 4300825984 00000100:00000040:3.0:1787698252.581546:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300825967, at 4300825984 00000100:00100000:0.0:1787698253.221556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698253.221556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698253.221556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698253.349632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698254.245532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698254.245532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698254.245650:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698254.373633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698255.269542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698255.269542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698255.269544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698255.397586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698255.781566:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698255.781580:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698255.781585:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698255.781598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6156 00000100:00080000:2.0:1787698255.781611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698255.781615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781619:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781624:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698255.781629:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698255.781633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781636:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781639:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698255.781643:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781670:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698255.781709:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698255.781727:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698255.781750:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698255.781753:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698255.781755:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698255.781759:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6156 00000100:00080000:2.0:1787698255.781763:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698255.781765:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781768:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781769:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698255.781772:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698255.781775:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781777:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781780:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698255.781782:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698255.781792:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698255.781797:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698255.781805:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698255.781813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126315776:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698255.781814:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698255.781822:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698255.781830:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698255.781841:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698255.781843:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698255.781846:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698255.781847:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6156 00000100:00080000:2.0:1787698255.781850:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698255.781852:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698255.781855:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781855:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781857:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698255.781859:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698255.781862:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781864:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781866:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698255.781868:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698255.781872:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698255.781875:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698255.781882:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698255.781890:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698255.781896:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698255.781906:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698255.781907:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698255.781914:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6156 00000100:00080000:2.0:1787698255.781917:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698255.781920:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781922:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781924:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698255.781927:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698255.781929:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.781932:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.781934:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698255.781937:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698255.781942:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698255.781946:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698255.781955:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698255.781963:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698255.781964:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698255.781972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315904:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698255.781973:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698255.781976:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698255.781978:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698255.781981:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6156 00000100:00080000:2.0:1787698255.781984:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698255.782001:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698255.782002:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698255.782005:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.782007:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698255.782009:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698255.782010:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698255.782012:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698255.782015:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698255.782017:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698255.782019:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698255.782024:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787698255.782025:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:2.0:1787698255.782029:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698255.782032:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698255.782040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698255.782055:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698255.782062:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698255.782078:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:0.0:1787698255.782086:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698255.782095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316160:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787698255.782113:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698255.782124:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698255.782129:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698255.782135:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698255.782135:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6166) 00000100:00000040:0.0:1787698255.782149:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:0.0:1787698255.782157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698255.782166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698255.782178:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698255.782186:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698255.782196:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698256.293516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698256.293516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698256.293516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698256.421636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698257.317526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698257.317526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698257.317526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698257.445591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698257.701547:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300831087, at 4300831104 00000100:00000040:3.0:1787698257.701547:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300831087, at 4300831104 00000100:00100000:0.0:1787698258.341537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698258.341537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698258.341537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698258.469625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698258.853806:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698258.853865:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698258.853884:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698258.853933:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698258.853972:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698258.853984:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698258.853995:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:0.0:1787698258.854005:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698255/real 1787698258] req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698258.854019:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698258.854038:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698258.854052:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698258.854055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698258.854076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698258.854082:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698258.854089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698258.854104:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698258.854116:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698280 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854120:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126315904:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787698258.854134:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698255/real 1787698258] req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698258.854160:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698258.854169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698258.854173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698258.854176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854182:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316032:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787698258.854188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698255/real 1787698258] req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698258.854206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698258.854212:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698258.854215:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698258.854225:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854232:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126315776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698258.854239:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126315904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698258.854252:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698258.854262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126316032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698258.854269:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698258.854275:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126315776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698258.854282:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698258.854290:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698255/real 1787698258] req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698258.854312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698258.854319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698258.854322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698258.854327:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854335:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316288:192.168.201.136@tcp:8:kworker.0 00000100:00100000:0.0:1787698258.854347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698255/real 1787698258] req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698258.854368:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698258.854375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698258.854378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698258.854381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698258.854389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698258.854394:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126316288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698258.854402:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698258.854407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126316160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698280 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698258.854414:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698259.365534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698259.365535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698259.365583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698259.493619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698260.389531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698260.389554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698260.389579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698260.517632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698260.901519:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698260.901528:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698260.901532:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698260.901543:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6161 00000100:00080000:2.0:1787698260.901651:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698260.901667:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901672:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901675:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698260.901680:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698260.901683:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901686:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901689:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698260.901692:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901707:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698260.901733:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698260.901748:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698260.901786:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698260.901791:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698260.901794:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698260.901799:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6161 00000100:00080000:2.0:1787698260.901805:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698260.901807:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901810:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901812:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698260.901815:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698260.901817:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901820:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901822:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698260.901825:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698260.901834:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698260.901839:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698260.901848:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698260.901857:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698260.901865:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698260.901868:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698260.901870:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698260.901874:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6161 00000100:00080000:2.0:1787698260.901877:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698260.901880:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901882:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901884:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698260.901886:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698260.901889:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901894:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698260.901896:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698260.901914:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698260.901919:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698260.901935:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698260.901946:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698260.901954:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698260.901958:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698260.901959:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698260.901963:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6161 00000100:00080000:2.0:1787698260.901967:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698260.901969:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901971:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901974:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698260.901976:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698260.901978:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.901981:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.901983:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698260.901986:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698260.901991:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698260.901995:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698260.902004:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698260.902014:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698260.902031:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698260.902035:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698260.902037:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698260.902040:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6161 00000100:00080000:2.0:1787698260.902044:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698260.902047:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.902050:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.902052:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698260.902054:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698260.902056:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698260.902059:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698260.902061:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698260.902063:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698260.902068:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698260.902072:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698260.902080:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698260.902089:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698260.902099:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6172) 00000100:00000040:0.0:1787698260.902231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.902246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126316416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698260.902281:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.902312:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698260.902353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698260.902373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.902415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698260/real 1787698260] req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.902484:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.902501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698260.902510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698260.902514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698260.902518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.902526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126316416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698260.902533:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.902542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698260.903464:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903473:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698260.903505:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698260.903519:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903539:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698260.903547:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698260/real 1787698260] req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903576:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698260.903586:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698260.903590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698260.903595:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903602:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698260.903607:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903616:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698260.903630:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903637:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698260.903650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698260.903657:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903665:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698260.903676:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698260/real 1787698260] req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903694:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698260.903701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698260.903704:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698260.903707:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903714:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698260.903718:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698260.903737:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903743:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698260.903754:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698260.903760:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903767:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698260.903772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698260/real 1787698260] req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903793:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903800:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698260.903807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698260.903810:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698260.903814:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903821:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698260.903827:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903835:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698260.903846:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903853:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698260.903864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698260.903870:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698285 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903878:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698260.903883:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698260/real 1787698260] req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903893:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698260.903924:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698260.903931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698260.903934:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698260.903938:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698260.903945:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126316928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698260.903951:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126316416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698260.903962:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698260.903970:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126316544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698260.903976:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698260.903981:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126316672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698260.903987:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698260.903992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126316800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698260.903998:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698260.904003:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126316928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698285 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698260.904010:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698261.413537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698261.413537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698261.413537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698261.541607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698262.437518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698262.437518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698262.437613:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698262.565531:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698262.821533:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300836207, at 4300836224 00000100:00000040:3.0:1787698262.821637:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300836207, at 4300836224 00000100:00100000:1.0:1787698263.461538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698263.462931:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698263.462951:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698263.589659:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698264.485517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698264.485517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698264.485517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698264.613553:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698265.509520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698265.509520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698265.509520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698265.637556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698266.021555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698266.021564:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698266.021568:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698266.021577:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6167 00000100:00080000:0.0:1787698266.021615:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698266.021619:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.021624:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021628:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698266.021633:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698266.021636:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.021638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021641:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698266.021644:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021666:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698266.021690:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698266.021705:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.021730:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698266.021733:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698266.021735:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698266.021739:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6167 00000100:00080000:0.0:1787698266.021743:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698266.021746:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.021762:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021765:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698266.021767:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698266.021771:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.021774:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021776:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698266.021780:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698266.021788:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698266.021789:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698266.021798:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698266.021826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126317056:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698266.021839:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698266.021852:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.021861:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698266.021864:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698266.021866:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698266.021869:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6167 00000100:00080000:0.0:1787698266.021874:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698266.021877:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698266.021882:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.021892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698266.021892:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021896:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698266.021899:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698266.021903:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.021906:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021908:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698266.021911:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698266.021923:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698266.021923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698266.021931:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698266.021941:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698266.021945:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698266.021968:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.021977:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698266.021981:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698266.021983:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698266.021989:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6167 00000100:00100000:1.0:1787698266.021989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.021996:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698266.022003:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.022007:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.022010:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698266.022012:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698266.022016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698266.022020:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.022028:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.022032:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698266.022034:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.022038:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:1.0:1787698266.022045:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787698266.022048:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698266.022051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698266.022055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698266.022057:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698266.022066:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126317056:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698266.022072:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698266.022073:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126317056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698266.022103:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698266.022108:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022120:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317184:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698266.022150:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698266.022155:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698266.022156:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698266.022161:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698266.022162:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6167 00000100:00080000:0.0:1787698266.022170:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698266.022177:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698266.022177:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698266.022184:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698266.022187:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698266.022189:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698266.022192:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698266.022198:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698266.022198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698266.022203:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698266.022207:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698266.022212:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698266.022213:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698266.022222:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698266.022228:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698266.022236:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698266.022238:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698266.022274:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698266.022285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698266.022292:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6177) 00000100:00080000:1.0:1787698266.022292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698266.022302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698266.022320:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698266.022341:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022348:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698266.022358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698266.022364:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022373:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698266.022378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698266.022404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698266.022407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698266.022410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698266.022448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126317184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698266.022457:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698266.022463:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126317312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698266.022477:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698266.022486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698266.022507:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022514:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698266.022524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698266.022530:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022539:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698266.022544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022563:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698266.022570:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698266.022573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698266.022576:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022584:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698266.022589:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022597:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698266.022605:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022611:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698266.022619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698266.022624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698291 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022632:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698266.022637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698266/real 1787698266] req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698266.022653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698266.022657:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698266.022658:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698266.022660:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698266.022664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698266.022668:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126317440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698266.022672:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698266.022675:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126317568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698291 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698266.022679:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698266.533525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698266.533525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698266.533526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698266.661548:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698267.557515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698267.557515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698267.557517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698267.685572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698267.941538:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300841327, at 4300841344 00000100:00000040:3.0:1787698267.941538:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300841327, at 4300841344 00000100:00100000:0.0:1787698268.581527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698268.581527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698268.581642:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698268.709542:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698269.605518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698269.605518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698269.605518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698269.733622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698270.629530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698270.629530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698270.630450:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698270.757558:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698271.141494:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698271.141501:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698271.141505:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698271.141512:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6172 00000100:00080000:0.0:1787698271.141521:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698271.141523:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141527:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141531:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698271.141535:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698271.141538:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141541:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141544:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698271.141548:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141562:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698271.141584:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698271.141595:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.141615:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698271.141619:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698271.141621:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698271.141625:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6172 00000100:00080000:0.0:1787698271.141630:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698271.141633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141637:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141639:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:1.0:1787698271.141652:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698271.141662:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00100000:1.0:1787698271.141665:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126317696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698271.141670:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141674:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141677:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698271.141680:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698271.141689:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698271.141694:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698271.141695:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.141704:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698271.141707:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698271.141715:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.141721:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698271.141735:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698271.141738:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698271.141741:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6172 00000800:00000100:1.0:1787698271.141745:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698271.141746:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698271.141750:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141753:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141755:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698271.141756:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698271.141759:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698271.141763:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698271.141765:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141771:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698271.141776:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698271.141782:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698271.141787:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698271.141795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.141807:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698271.141814:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698271.141826:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.141828:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.141847:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698271.141848:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698271.141853:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698271.141855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698271.141856:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698271.141860:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.141862:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6172 00000100:00080000:0.0:1787698271.141867:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698271.141873:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698271.141873:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126317696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698271.141876:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141880:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698271.141882:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698271.141883:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126317696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698271.141886:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141895:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698271.141900:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698271.141902:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:0.0:1787698271.141904:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698271.141909:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698271.141916:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698271.141918:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698271.141935:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.141937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317824:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698271.141950:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698271.141952:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:1.0:1787698271.141955:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698271.141962:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698271.141966:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698271.141969:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6172 00000100:00080000:0.0:1787698271.141976:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698271.141979:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698271.141980:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141983:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141985:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698271.141987:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698271.141989:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698271.141991:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698271.141996:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698271.141999:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698271.142003:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698271.142008:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000008:00100000:0.0:1787698271.142009:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698271.142015:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698271.142016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698271.142030:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698271.142030:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698271.142063:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698271.142064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698271.142070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698271.142077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698271.142082:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6182) 00000100:00100000:1.0:1787698271.142084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698271.142089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698271.142101:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142106:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698271.142111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698271.142114:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142119:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698271.142122:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142132:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698271.142141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698271.142143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698271.142145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126317952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698271.142151:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126317824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698271.142156:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698271.142159:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126317952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698271.142163:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698271.142168:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698271.142179:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142183:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698271.142188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698271.142191:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142197:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698271.142200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698271.142214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698271.142216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698271.142218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698271.142224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142228:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698271.142233:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142237:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698271.142242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698271.142247:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698296 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142261:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698271.142266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698271/real 1787698271] req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698271.142284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698271.142291:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698271.142294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698271.142298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698271.142305:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698271.142310:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126318080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698271.142318:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698271.142322:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126318208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698296 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698271.142326:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698271.653535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698271.653543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698271.653608:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698271.781616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698272.677517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698272.677517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698272.677517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698272.805633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698273.061498:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300846447, at 4300846464 00000100:00000040:3.0:1787698273.061511:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300846447, at 4300846464 00000100:00100000:1.0:1787698273.701513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698273.701513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698273.701638:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698273.829588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698274.725549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698274.725549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698274.725570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698274.853624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698275.749489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698275.749495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698275.749525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698275.877633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698276.261565:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698276.261578:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698276.261583:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698276.261596:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6177 00000100:00080000:0.0:1787698276.261609:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698276.261613:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698276.261618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.261658:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698276.261663:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698276.261668:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698276.261671:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.261674:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698276.261678:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698276.261693:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698276.261725:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698276.261742:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698276.261768:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698276.261772:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698276.261774:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698276.261779:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6177 00000100:00080000:0.0:1787698276.261783:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698276.261787:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698276.261791:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.261793:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698276.261797:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698276.261800:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698276.261803:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.261807:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787698276.261809:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698276.261812:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698276.261825:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698276.261828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126318336:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698276.261833:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698276.261860:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698276.261860:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.261885:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698276.261893:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698276.261905:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698276.261908:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698276.261911:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698276.261915:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6177 00000100:00080000:0.0:1787698276.261921:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698276.261926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698276.261929:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698276.261934:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.261936:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698276.261939:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698276.261943:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698276.261947:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698276.261950:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.261957:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698276.261962:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698276.261968:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698276.261974:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698276.261986:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698276.261988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698276/real 1787698276] req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698276.262019:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698276.262030:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698276.262039:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:1.0:1787698276.262044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698276.262049:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698276.262053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698276.262060:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6177 00000100:00080000:1.0:1787698276.262060:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698276.262065:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698276.262068:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698276.262077:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698276.262082:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126318336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698276.262086:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.262090:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698276.262094:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698276.262095:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126318336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698276.262101:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698276.262109:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.262113:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000020:00000040:1.0:1787698276.262116:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787698276.262117:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698276.262126:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698276.262132:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698276.262137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698276.262145:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698276.262149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698276.262166:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262169:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262196:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698276.262198:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698276.262203:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698276.262205:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698276.262206:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698276.262215:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6177 00000100:00000040:1.0:1787698276.262216:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698276.262223:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698276.262227:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:0.0:1787698276.262230:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698276.262235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698276/real 1787698276] req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698276.262239:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698276.262245:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698276.262250:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698276.262252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698276.262265:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698276.262265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698276.262275:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698276.262277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698276.262281:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698276.262283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698276.262290:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698276.262290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698276.262301:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698276.262308:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698276.262312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318464:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787698276.262320:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698276.262322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698276.262349:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698276.262364:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6187) 00000100:00000040:1.0:1787698276.262365:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262375:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698276.262394:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698276.262402:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262412:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698276.262438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698276/real 1787698276] req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262454:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698276.262459:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698276.262461:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698276.262463:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698276.262475:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126318464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698276.262480:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698276.262485:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126318592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698276.262489:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698276.262495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698276.262513:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262520:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698276.262531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698276.262538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262546:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698276.262552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698276/real 1787698276] req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262570:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698276.262578:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698276.262581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698276.262584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262591:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698276.262597:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262603:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698276.262620:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262625:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698276.262633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698276.262652:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698301 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262661:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698276.262665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698276/real 1787698276] req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262674:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698276.262680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698276.262685:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698276.262688:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698276.262696:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698276.262702:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126318848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698276.262706:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126318720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698276.262711:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698276.262715:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126318848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698301 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698276.262720:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698276.773507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698276.773509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698276.773513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698276.901603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698277.797505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698277.797505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698277.797506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698277.925626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698278.181514:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300851567, at 4300851584 00000100:00000040:3.0:1787698278.181514:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300851567, at 4300851584 00000100:00100000:1.0:1787698278.821505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698278.821507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698278.821523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698278.949536:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698279.845491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698279.845491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698279.845491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698279.973647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698280.869492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698280.869495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698280.869556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698280.997634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698281.381534:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698281.381542:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698281.381545:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698281.381563:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6182 00000100:00080000:1.0:1787698281.381571:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698281.381574:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381576:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381579:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698281.381583:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698281.381585:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381587:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381589:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698281.381591:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381601:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698281.381619:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698281.381630:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698281.381648:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698281.381651:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698281.381652:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698281.381655:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6182 00000100:00080000:1.0:1787698281.381657:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698281.381658:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381660:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381661:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698281.381663:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698281.381665:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381667:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381669:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698281.381670:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698281.381676:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698281.381680:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698281.381685:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698281.381689:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698281.381709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126318976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698281.381709:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698281.381720:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698281.381722:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698281.381723:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698281.381725:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6182 00000100:00080000:1.0:1787698281.381727:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698281.381728:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381730:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381731:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698281.381732:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698281.381734:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381735:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381736:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698281.381737:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698281.381741:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698281.381743:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698281.381744:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698281.381752:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698281.381764:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.381765:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698281.381772:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698281.381775:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698281.381777:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698281.381779:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6182 00000100:00080000:1.0:1787698281.381791:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698281.381793:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381794:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381796:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698281.381797:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698281.381799:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381800:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381801:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698281.381804:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:0.0:1787698281.381807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787698281.381809:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698281.381813:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698281.381820:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698281.381825:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698281.381831:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698281.381838:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698281.381840:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698281.381841:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698281.381844:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6182 00000100:00080000:1.0:1787698281.381845:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698281.381847:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381849:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381850:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698281.381851:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698281.381854:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698281.381855:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698281.381856:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698281.381858:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698281.381861:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698281.381863:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698281.381868:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698281.381875:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698281/real 1787698281] req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698281.381876:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698281.381894:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6192) 00000100:00100000:0.0:1787698281.381973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.381995:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698281.382001:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698281.382003:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698281.382006:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126318976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698281.382016:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126318976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698281.382022:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698281.382034:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382039:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698281.382048:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382053:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698281.382062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698281.382066:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382073:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698281.382076:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698281/real 1787698281] req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382096:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698281.382101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698281.382102:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698281.382104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382108:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698281.382111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698281.382123:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382127:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698281.382134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698281.382137:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382143:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698281.382146:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698281/real 1787698281] req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698281.382171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698281.382173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698281.382174:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698281.382181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698281.382192:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382196:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698281.382203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698281.382206:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382212:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698281.382214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698281/real 1787698281] req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698281.382237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698281.382238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698281.382240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698281.382247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698281.382257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382261:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698281.382266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698281.382270:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698306 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382275:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698281.382277:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698281/real 1787698281] req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698281.382296:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698281.382303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698281.382306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698281.382309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698281.382316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698281.382326:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126319104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698281.382333:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698281.382338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126319232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698281.382344:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698281.382349:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126319360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698281.382354:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698281.382359:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126319488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698306 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698281.382364:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698281.893508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698281.893508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698281.893508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698282.021586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698282.917470:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698282.917484:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698282.917490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698283.045561:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698283.301475:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300856687, at 4300856704 00000100:00000040:2.0:1787698283.301484:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300856687, at 4300856704 00000100:00100000:2.0:1787698283.941513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698283.941516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698283.941516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698284.069651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698284.965587:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698284.965601:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698284.965602:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698285.093669:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698285.989562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698285.989569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698285.989652:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698286.117630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698286.501553:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698286.501566:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698286.501570:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698286.501584:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6187 00000100:00080000:1.0:1787698286.501598:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698286.501602:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501606:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.501611:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698286.501617:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698286.501620:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501623:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.501626:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698286.501630:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698286.501647:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698286.501680:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698286.501696:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698286.501722:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698286.501725:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698286.501727:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698286.501732:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6187 00000100:00080000:1.0:1787698286.501736:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698286.501740:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501744:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.501746:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698286.501750:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698286.501753:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501758:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698286.501760:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698286.501762:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698286.501769:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698286.501781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126319616:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787698286.501783:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698286.501790:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698286.501803:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698286.501813:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698286.501816:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.501829:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698286.501838:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698286.501842:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698286.501844:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698286.501849:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6187 00000100:00080000:1.0:1787698286.501854:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698286.501858:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501860:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.501863:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698286.501866:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:0.0:1787698286.501870:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501873:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501877:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.501880:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698286.501884:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698286.501889:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698286.501890:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698286.501899:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698286.501912:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698286.501923:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.501928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698286/real 1787698286] req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698286.501939:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698286.501943:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698286.501946:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698286.501955:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6187 00000100:00080000:1.0:1787698286.501960:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698286.501965:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698286.501965:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501972:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.501978:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698286.501981:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698286.501987:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698286.501988:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.501996:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698286.502002:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787698286.502002:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698286.502007:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:0.0:1787698286.502009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698286.502015:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126319616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698286.502026:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126319616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698286.502030:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698286.502035:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:1.0:1787698286.502037:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698286.502048:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698286.502063:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502070:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698286.502077:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698286.502081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319744:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698286.502084:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698286.502088:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698286.502094:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6187 00000100:00080000:1.0:1787698286.502099:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698286.502103:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698286.502105:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.502111:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.502115:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698286.502119:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698286.502125:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698286.502128:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698286.502132:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698286.502135:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000800:00000100:0.0:1787698286.502141:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698286.502141:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698286.502149:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787698286.502151:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698286.502158:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698286.502162:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698286.502168:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698286/real 1787698286] req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698286.502172:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698286.502189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698286.502192:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698286.502217:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6197) 00000100:00080000:0.0:1787698286.502226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698286.502229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698286.502232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698286.502241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698286.502265:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502270:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698286.502278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698286.502282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502289:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698286.502292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698286/real 1787698286] req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698286.502309:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698286.502310:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698286.502312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502317:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126319872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698286.502320:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126319744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698286.502325:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698286.502333:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126319872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698286.502338:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698286.502342:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698286.502357:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502361:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698286.502367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698286.502371:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502377:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698286.502380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698286/real 1787698286] req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698286.502396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698286.502398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698286.502400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502404:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698286.502412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698286.502445:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502449:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698286.502456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698286.502460:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698311 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502478:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698286.502482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698286/real 1787698286] req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698286.502495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698286.502499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698286.502500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698286.502502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698286.502507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698286.502510:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126320000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698286.502523:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698286.502527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126320128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698311 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698286.502531:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698287.013511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698287.013516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698287.013558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698287.141595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698288.037498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698288.037548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698288.037573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698288.165562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698288.421494:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300861807, at 4300861824 00000100:00000040:2.0:1787698288.421496:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300861807, at 4300861824 00000100:00100000:0.0:1787698289.061557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698289.061572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698289.061723:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698289.189566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698290.085516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698290.085516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698290.085535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698290.213568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698291.109495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698291.109500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698291.109502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698291.237577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698291.621517:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698291.621527:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698291.621531:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698291.621542:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6192 00000100:00080000:3.0:1787698291.621552:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698291.621556:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698291.621560:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621564:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698291.621568:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698291.621571:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698291.621574:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621576:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698291.621579:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621592:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698291.621617:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698291.621632:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698291.621654:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698291.621657:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698291.621659:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698291.621679:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6192 00000100:00080000:3.0:1787698291.621684:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698291.621687:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698291.621701:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698291.621704:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621710:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698291.621715:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698291.621720:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698291.621724:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126320256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698291.621727:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621732:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698291.621735:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698291.621744:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698291.621750:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698291.621764:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698291.621764:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698291.621797:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698291.621798:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698291.621809:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698291.621813:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698291.621815:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698291.621819:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6192 00000100:00080000:3.0:1787698291.621822:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698291.621826:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698291.621829:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621831:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698291.621833:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698291.621836:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:0.0:1787698291.621840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698291.621850:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621854:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698291.621857:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698291.621862:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698291.621864:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698291.621871:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698291.621885:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698291.621896:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.621904:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698291/real 1787698291] req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698291.621907:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698291.621915:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698291.621919:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698291.621924:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6192 00000100:00100000:0.0:1787698291.621927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698291.621930:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698291.621937:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698291.621939:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698291.621945:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.621950:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698291.621952:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698291.621958:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698291.621965:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698291.621969:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787698291.621970:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698291.621976:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698291.621978:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698291.621984:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698291.621985:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698291.621993:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698291.622002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126320256:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698291.622008:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698291.622014:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126320256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698291.622042:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698291.622043:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698291.622055:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698291.622059:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698291.622061:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698291.622065:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698291.622068:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6192 00000100:00080000:3.0:1787698291.622075:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698291.622081:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698291.622084:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698291.622089:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.622093:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698291.622096:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698291.622099:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698291.622101:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698291.622105:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698291.622112:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787698291.622116:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698291.622120:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698291.622127:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698291.622132:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:0.0:1787698291.622135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698291.622145:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698291.622151:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698291.622165:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622168:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698291.622180:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698291/real 1787698291] req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698291.622182:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6202) 00000100:00100000:0.0:1787698291.622198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698291.622235:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698291.622237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698291.622239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698291.622247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698291.622270:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622278:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698291.622290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698291.622297:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622306:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698291.622310:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698291/real 1787698291] req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698291.622332:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698291.622334:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698291.622336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622341:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698291.622344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698291.622357:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622362:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698291.622369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698291.622373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622400:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698291.622404:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698291/real 1787698291] req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622416:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698291.622445:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698291.622451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698291.622454:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698291.622462:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126320384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698291.622467:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698291.622471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126320512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698291.622476:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698291.622478:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126320640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698291.622482:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698291.622487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622492:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698291.622502:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622509:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698291.622521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698291.622527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698316 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622537:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698291.622543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698291/real 1787698291] req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698291.622567:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698291.622574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698291.622576:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698291.622579:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698291.622586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126320768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698291.622592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126320768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698316 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698291.622599:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698292.133495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698292.133496:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698292.133497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698292.261564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698293.157492:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698293.157492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698293.157492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698293.285577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698293.541528:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300866927, at 4300866944 00000100:00000040:3.0:1787698293.541528:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300866927, at 4300866944 00000100:00100000:1.0:1787698294.181508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698294.181514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698294.181514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698294.309601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698295.205505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698295.205506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698295.205539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698295.333601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698296.229512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698296.229512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698296.229512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698296.357542:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698296.741515:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698296.741523:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698296.741526:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698296.741533:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6197 00000100:00080000:3.0:1787698296.741542:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698296.741544:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741547:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741550:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698296.741554:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698296.741556:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741558:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741560:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698296.741562:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741573:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698296.741592:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698296.741603:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698296.741622:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698296.741625:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698296.741626:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698296.741633:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6197 00000100:00080000:3.0:1787698296.741636:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698296.741650:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741652:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741654:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698296.741656:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698296.741659:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741661:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741662:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698296.741667:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698296.741668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698296.741676:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698296.741682:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126320896:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698296.741684:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698296.741699:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698296.741711:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698296.741713:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.741737:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698296.741740:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698296.741744:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698296.741748:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698296.741754:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6197 00000100:00080000:3.0:1787698296.741759:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698296.741762:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741765:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741768:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698296.741770:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698296.741773:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741775:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741776:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000800:00000100:0.0:1787698296.741781:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698296.741787:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698296.741792:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698296.741795:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698296.741802:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698296.741805:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698296.741829:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698296.741836:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698296.741839:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698296.741841:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698296.741845:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6197 00000100:00080000:3.0:1787698296.741849:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787698296.741854:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698296/real 1787698296] req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698296.741866:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698296.741873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698296.741873:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741879:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698296.741882:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698296.741888:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741890:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698296.741894:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698296.741894:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698296.741900:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698296.741907:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698296.741910:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787698296.741912:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698296.741917:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:3.0:1787698296.741925:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698296.741926:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698296.741946:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.741949:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126320896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698296.741961:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126320896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698296.741962:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698296.741969:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698296.741972:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698296.741978:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6197 00000020:00000040:0.0:1787698296.741981:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698296.741983:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698296.741988:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.741991:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.741993:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698296.741996:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698296.742001:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698296.742001:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698296.742008:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698296.742013:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698296.742018:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698296.742022:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321024:192.168.201.136@tcp:38:kworker.0 00000008:00100000:3.0:1787698296.742025:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698296.742030:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698296.742037:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698296.742041:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698296.742051:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698296.742058:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698296.742065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698296.742068:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6207) 00000100:00000040:0.0:1787698296.742080:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742088:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698296.742094:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698296/real 1787698296] req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742101:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742107:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698296.742111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698296.742113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698296.742115:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698296.742122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698296.742135:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742139:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698296.742145:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698296.742149:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742162:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698296.742165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698296/real 1787698296] req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698296.742182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698296.742183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698296.742185:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698296.742192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698296.742205:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742209:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698296.742215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698296.742218:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698296.742231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698296/real 1787698296] req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742238:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698296.742247:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698296.742249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698296.742251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698296.742258:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126321024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698296.742263:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698296.742266:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126321152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698296.742270:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698296.742272:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126321280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698296.742277:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698296.742281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742286:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698296.742320:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742325:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698296.742332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698296.742336:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698321 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742342:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698296.742345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698296/real 1787698296] req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698296.742357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698296.742361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698296.742362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698296.742364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698296.742368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698296.742371:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126321408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698321 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698296.742375:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698297.253535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698297.253536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698297.253536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698297.381605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698298.277505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698298.277506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698298.277507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698298.405562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698298.661480:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300872047, at 4300872064 00000100:00000040:3.0:1787698298.661480:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300872047, at 4300872064 00000100:00100000:0.0:1787698299.301510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698299.301510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698299.301510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698299.429588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698300.325496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698300.325496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698300.325501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698300.453558:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698301.349490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698301.349492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698301.349494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698301.477584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698301.861538:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698301.861548:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698301.861551:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698301.861561:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6202 00000100:00080000:0.0:1787698301.861571:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698301.861575:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.861579:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.861582:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698301.861587:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698301.861590:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.861592:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.861595:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698301.861598:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698301.861612:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698301.861636:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698301.861658:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698301.861682:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698301.861686:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698301.861688:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698301.861692:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6202 00000100:00080000:0.0:1787698301.861696:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698301.861700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.861717:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.861719:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698301.861722:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698301.861730:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698301.861733:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698301.861737:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.861749:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698301.861755:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698301.861760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126321536:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787698301.861765:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698301.861770:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698301.861780:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698301.861788:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.861794:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.861821:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698301.861824:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698301.861829:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698301.861832:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698301.861837:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6202 00000100:00080000:0.0:1787698301.861848:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698301.861851:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.861853:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.861854:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698301.861856:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:1.0:1787698301.861862:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698301.861870:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.861873:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.861876:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698301.861879:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698301.861887:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698301.861896:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698301.861906:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698301.861923:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698301.861924:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698301/real 1787698301] req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.861966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698301.861971:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.861998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698301.862001:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698301.862013:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698301.862016:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698301.862025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698301.862026:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6202 00000100:00080000:0.0:1787698301.862030:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698301.862033:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.862035:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.862036:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698301.862038:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698301.862040:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.862041:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.862043:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698301.862044:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698301.862048:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698301.862052:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698301.862061:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698301.862065:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698301.862077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698301.862082:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126321536:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698301.862148:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698301.862150:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126321536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698301.862157:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698301.862161:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698301.862171:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6202 00000020:00000040:1.0:1787698301.862176:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787698301.862177:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698301.862185:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.862189:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.862192:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698301.862194:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698301.862198:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698301.862218:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698301.862220:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698301.862221:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698301.862225:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698301.862228:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698301.862235:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698301.862241:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698301.862249:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6212) 00000100:00000040:1.0:1787698301.862286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698301.862316:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862321:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698301.862331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698301.862336:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698301.862347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698301/real 1787698301] req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698301.862368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698301.862371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698301.862373:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698301.862381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862386:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698301.862393:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862398:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698301.862403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698301.862407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862412:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698301.862415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698301/real 1787698301] req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862451:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698301.862457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698301.862459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698301.862461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698301.862469:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698301.862482:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862487:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698301.862493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698301.862496:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862502:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698301.862505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698301/real 1787698301] req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862517:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698301.862521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698301.862522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698301.862524:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862529:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126321920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698301.862537:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126321664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698301.862542:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698301.862547:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126321792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698301.862551:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698301.862554:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126321920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698301.862558:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698301.862564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698301.862575:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862579:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698301.862595:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698301.862602:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698326 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862611:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698301.862617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698301/real 1787698301] req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698301.862642:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698301.862646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698301.862648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698301.862650:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698301.862654:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698301.862658:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126322048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698326 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698301.862662:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698302.373518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698302.373518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698302.373545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698302.501644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698303.397560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698303.397560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698303.398244:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698303.525563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698303.781488:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300877167, at 4300877184 00000100:00000040:2.0:1787698303.781514:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300877167, at 4300877184 00000100:00100000:1.0:1787698304.421533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698304.421533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698304.421540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698304.549541:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698305.445512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698305.445534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698305.445535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698305.574607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698306.469615:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698306.469737:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698306.469997:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698306.597556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698306.981516:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698306.981523:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698306.981526:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698306.981533:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6207 00000100:00080000:0.0:1787698306.981542:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698306.981544:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981547:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981550:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698306.981554:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698306.981556:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981558:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981560:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698306.981562:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981573:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698306.981593:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698306.981603:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698306.981621:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698306.981623:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698306.981624:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698306.981627:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6207 00000100:00080000:0.0:1787698306.981629:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698306.981631:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981632:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981634:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698306.981649:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698306.981652:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981655:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981656:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698306.981658:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698306.981665:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698306.981669:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698306.981675:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698306.981679:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698306.981703:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698306.981725:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698306.981730:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698306.981734:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698306.981735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126322176:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698306.981739:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6207 00000100:00080000:0.0:1787698306.981744:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698306.981746:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981748:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981749:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698306.981751:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698306.981753:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981755:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981756:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698306.981758:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698306.981762:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698306.981765:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698306.981769:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.981781:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698306.981787:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698306.981794:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698306.981801:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698306.981803:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698306.981804:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698306.981807:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6207 00000100:00080000:0.0:1787698306.981810:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698306.981812:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981813:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698306.981816:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698306.981820:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981823:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698306.981823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981828:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698306.981831:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698306.981837:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698306.981842:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698306.981851:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698306.981855:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698306.981877:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698306.981886:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698306.981889:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698306.981897:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698306.981903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698306/real 1787698306] req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698306.981906:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6207 00000100:00080000:0.0:1787698306.981913:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698306.981919:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698306.981922:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981924:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698306.981926:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698306.981930:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698306.981932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698306.981933:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698306.981936:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698306.981940:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698306.981946:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698306.981951:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698306.981954:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698306.981962:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698306.981973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698306.981975:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698306.981982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698306.981987:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6218) 00000100:00000040:1.0:1787698306.981991:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126322176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698306.982009:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126322176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698306.982020:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698306.982081:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698306.982140:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982148:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698306.982163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698306.982171:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982194:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698306.982200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698306/real 1787698306] req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982213:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982223:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698306.982231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698306.982234:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698306.982238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982246:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698306.982252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698306.982273:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982280:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698306.982293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698306.982300:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982310:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698306.982316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698306/real 1787698306] req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982328:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698306.982346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698306.982349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698306.982353:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982362:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698306.982368:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982377:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698306.982390:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982398:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698306.982409:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698306.982416:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982450:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698306.982457:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698306/real 1787698306] req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982469:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982478:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698306.982485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698306.982489:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698306.982492:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982500:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698306.982505:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698306.982526:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982534:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698306.982546:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698306.982552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698331 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982560:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698306.982565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698306/real 1787698306] req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698306.982585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698306.982592:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698306.982595:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698306.982599:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698306.982607:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698306.982612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126322304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698306.982621:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698306.982627:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126322432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698306.982635:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698306.982641:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126322560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698306.982647:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698306.982653:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126322688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698331 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698306.982660:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698307.493609:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698307.494627:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698307.494768:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698307.621616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698308.517514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698308.517515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698308.517515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698308.645566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698308.901566:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300882287, at 4300882304 00000100:00000040:3.0:1787698308.901579:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300882287, at 4300882304 00000100:00100000:1.0:1787698309.541585:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698309.541586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698309.541595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698309.669731:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698310.565535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698310.565535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698310.565544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698310.693592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698311.589547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698311.589548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698311.589569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698311.717635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698312.101720:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698312.101733:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698312.101738:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698312.101752:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6213 00000100:00080000:2.0:1787698312.101765:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698312.101770:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.101775:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.101779:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698312.101785:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698312.101788:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.101791:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.101794:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698312.101798:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698312.101816:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698312.101848:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698312.101864:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698312.101890:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698312.101894:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698312.101896:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698312.101900:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6213 00000100:00080000:2.0:1787698312.101903:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698312.101906:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.101909:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.101913:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698312.101916:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698312.101920:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.101923:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.101926:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698312.101929:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698312.101939:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698312.101946:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698312.101947:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698312.101958:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698312.101970:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126322816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698312.101974:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698312.101985:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698312.101988:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698312.101989:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698312.101993:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6213 00000100:00080000:2.0:1787698312.101996:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698312.101999:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.102002:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.102004:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698312.102007:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698312.102011:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698312.102015:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.102021:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698312.102042:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698312.102048:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698312.102055:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698312.102060:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698312.102067:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698312.102076:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698312.102086:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698312.102094:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698312.102099:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787698312.102100:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698312.102102:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698312.102109:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6213 00000100:00080000:2.0:1787698312.102114:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698312.102117:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.102120:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.102122:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698312.102124:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698312.102128:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698312.102130:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698312.102136:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.102141:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698312.102147:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698312.102152:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698312.102158:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698312.102166:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698312.102177:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698312.102188:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698312.102196:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698312.102200:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698312.102208:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6213 00000100:00080000:2.0:1787698312.102212:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698312.102217:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698312.102218:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698312.102225:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698312.102230:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787698312.102232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698312.102234:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:1.0:1787698312.102243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698312.102243:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698312.102251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698312.102251:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698312.102257:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698312.102257:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698312.102266:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698312.102268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126322816:192.168.201.136@tcp:250:kworker.0 00000008:00100000:2.0:1787698312.102274:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698312.102275:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126322816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698312.102283:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787698312.102294:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698312.102299:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698312.102311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698312.102313:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698312.102333:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6223) 00000100:00000040:1.0:1787698312.102355:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102363:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698312.102375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698312.102382:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102391:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698312.102397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102406:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.102460:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698312.102463:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698312.102466:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126322944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698312.102478:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698312.102498:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102505:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698312.102515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698312.102520:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102525:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698312.102529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102540:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.102546:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698312.102548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698312.102550:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698312.102557:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698312.102569:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102573:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698312.102578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698312.102582:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102587:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698312.102590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102601:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.102604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698312.102607:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698312.102609:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102614:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698312.102616:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126322944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698312.102621:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698312.102624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126323072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698312.102628:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698312.102631:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126323200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698312.102634:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698312.102639:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698312.102651:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102655:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698312.102660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698312.102664:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698337 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102669:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698312.102672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698312/real 1787698312] req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102681:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698312.102689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698312.102696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698312.102699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698312.102703:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698312.102711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698312.102717:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126323328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698337 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698312.102724:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698312.613544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698312.613565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698312.613569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698312.741656:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698313.637525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698313.637525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698313.637577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698313.765542:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698314.021520:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300887407, at 4300887424 00000100:00000040:2.0:1787698314.021523:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300887407, at 4300887424 00000100:00100000:3.0:1787698314.661490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698314.661495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698314.661538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698314.789572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698315.685527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698315.685527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698315.685638:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698315.813569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698316.709493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698316.709516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698316.709563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698316.837620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698317.221548:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698317.221557:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698317.221561:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698317.221568:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6218 00000100:00080000:1.0:1787698317.221577:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698317.221579:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221582:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221585:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698317.221588:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698317.221590:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221592:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221594:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698317.221596:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221609:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698317.221631:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698317.221646:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698317.221668:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698317.221671:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698317.221672:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698317.221675:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6218 00000100:00080000:1.0:1787698317.221677:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698317.221679:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221681:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221682:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698317.221684:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698317.221686:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221688:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221690:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698317.221692:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698317.221699:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698317.221702:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698317.221710:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698317.221716:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698317.221722:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698317.221725:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698317.221729:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698317.221733:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698317.221733:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6218 00000100:00080000:1.0:1787698317.221739:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698317.221744:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221751:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221756:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698317.221761:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698317.221770:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221776:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221777:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698317.221778:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698317.221782:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698317.221786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126323456:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698317.221787:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698317.221797:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698317.221810:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698317.221823:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698317.221826:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698317.221828:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698317.221833:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6218 00000100:00080000:1.0:1787698317.221837:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698317.221840:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221844:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698317.221850:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698317.221850:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698317.221855:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698317.221868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221878:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221882:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:0.0:1787698317.221883:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698317.221890:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698317.221895:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698317.221900:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698317.221908:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698317.221915:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698317.221920:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698317.221923:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698317.221924:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698317.221927:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6218 00000100:00080000:1.0:1787698317.221930:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698317.221932:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221934:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221935:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698317.221950:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:0.0:1787698317.221953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221957:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698317.221961:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698317.221963:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698317.221964:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698317.221969:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698317.221972:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698317.221978:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698317.221985:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698317.221997:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698317.222009:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6228) 00000100:00100000:0.0:1787698317.222040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698317/real 1787698317] req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222070:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698317.222080:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698317.222084:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698317.222088:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126323456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698317.222104:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126323456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698317.222116:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698317.222134:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698317.222174:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222181:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698317.222195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698317.222202:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222213:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698317.222218:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698317/real 1787698317] req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698317.222245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698317.222248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698317.222252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698317.222266:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698317.222287:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222301:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698317.222313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698317.222320:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222330:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698317.222336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698317/real 1787698317] req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222346:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698317.222362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698317.222365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698317.222369:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698317.222382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698317.222402:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222409:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698317.222457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698317.222464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222474:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698317.222479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698317/real 1787698317] req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222489:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698317.222504:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698317.222507:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698317.222510:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698317.222522:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698317.222548:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222555:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698317.222564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698317.222571:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698342 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222593:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698317.222599:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698317/real 1787698317] req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698317.222617:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698317.222624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698317.222626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698317.222630:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698317.222637:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126323968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698317.222641:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126323584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698317.222647:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698317.222652:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126323712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698317.222658:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698317.222663:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126323840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698317.222668:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698317.222672:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126323968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698342 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698317.222682:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698317.733503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698317.733503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698317.733503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698317.861538:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698318.757544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698318.757544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698318.757544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698318.885595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698319.141549:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300892527, at 4300892544 00000100:00000040:2.0:1787698319.141592:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300892527, at 4300892544 00000100:00100000:1.0:1787698319.781509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698319.781509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698319.781511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698319.909602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698320.805521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698320.805525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698320.805575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698320.933610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698321.829532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698321.829610:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698321.829756:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698321.957555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698322.341494:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698322.341507:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698322.341513:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698322.341527:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6223 00000100:00080000:1.0:1787698322.341542:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698322.341547:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341554:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341558:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698322.341565:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698322.341569:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341574:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341578:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698322.341582:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341596:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698322.341628:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698322.341646:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698322.341667:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698322.341671:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698322.341673:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698322.341678:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6223 00000100:00080000:1.0:1787698322.341682:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698322.341686:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341691:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698322.341693:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698322.341694:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698322.341698:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698322.341705:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341715:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00100000:0.0:1787698322.341721:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126324096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698322.341721:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698322.341736:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698322.341743:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698322.341755:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698322.341759:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698322.341776:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698322.341777:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698322.341794:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698322.341798:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698322.341799:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698322.341804:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6223 00000100:00080000:1.0:1787698322.341808:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698322.341811:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341814:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341816:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698322.341819:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698322.341822:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341827:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698322.341830:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698322.341835:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698322.341840:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698322.341869:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698322.341884:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698322.341889:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698322.341904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324224:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698322.341906:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698322.341923:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:0.0:1787698322.341926:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698322.341926:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698322.341935:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6223 00000100:00080000:1.0:1787698322.341943:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698322.341944:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341949:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341953:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341955:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698322.341957:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698322.341960:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.341963:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.341966:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698322.341969:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698322.341974:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000008:00100000:1.0:1787698322.341975:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698322.341983:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698322.341987:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698322.341997:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698322.342000:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698322.342016:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698322.342021:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698322.342037:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787698322.342040:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698322.342043:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698322.342046:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698322.342051:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6223 00000100:00000040:0.0:1787698322.342056:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00080000:1.0:1787698322.342056:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698322.342062:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.342067:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.342069:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:0.0:1787698322.342071:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698322.342072:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698322.342077:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698322.342079:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698322.342082:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698322.342085:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698322.342090:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698322.342095:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698322.342108:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324480:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698322.342118:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698322.342129:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698322.342132:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698322.342152:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698322.342155:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6233) 00000100:00000040:0.0:1787698322.342169:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:0.0:1787698322.342181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698322.342191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698322.342218:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698322.342226:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698322.342236:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698322.853474:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698322.853474:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698322.853480:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698322.981518:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698323.877474:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698323.877513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698323.877570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698324.005608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698324.261510:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300897647, at 4300897664 00000100:00000040:2.0:1787698324.261513:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300897647, at 4300897664 00000100:00100000:1.0:1787698324.901626:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698324.901627:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698324.902126:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698325.029606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698325.413741:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698325.413823:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698325.413843:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698325.413884:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698325.413907:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698325.413918:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.413939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698322/real 1787698325] req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698325.413953:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698325.413966:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.413971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698325.413987:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698325.413992:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698325.413999:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698325.414012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698325.414020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000800:00000100:3.0:1787698325.414021:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698325.414029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698325.414034:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698347 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324224:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787698325.414072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698322/real 1787698325] req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698325.414097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698325.414105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698325.414108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698325.414111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126324096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698325.414125:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126324224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698325.414140:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698325.414151:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126324096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698325.414158:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698325.414166:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698322/real 1787698325] req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414178:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698325.414192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698325.414200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698325.414204:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698325.414209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414217:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324608:192.168.201.136@tcp:8:kworker.0 00000100:00100000:0.0:1787698325.414223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698322/real 1787698325] req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698325.414244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698325.414251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698325.414254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698325.414257:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324480:192.168.201.136@tcp:8:kworker.0 00000100:00100000:0.0:1787698325.414272:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698322/real 1787698325] req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414297:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698325.414305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698325.414314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698325.414317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698325.414321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698325.414328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698325.414334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126324608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698325.414343:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698325.414350:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126324480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698325.414358:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698325.414362:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126324352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698347 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698325.414370:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698325.925535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698325.925535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698325.925535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698326.053613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698326.949536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698326.949536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698326.949536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698327.077596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698327.461523:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698327.461533:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698327.461542:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698327.461555:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6228 00000100:00080000:1.0:1787698327.461566:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698327.461568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461574:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698327.461579:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698327.461581:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461583:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461585:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698327.461587:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461598:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698327.461624:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698327.461636:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698327.461661:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698327.461665:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698327.461666:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698327.461671:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6228 00000100:00080000:1.0:1787698327.461674:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698327.461676:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461679:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461681:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698327.461684:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698327.461686:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461689:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461691:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698327.461694:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698327.461704:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698327.461709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698327.461719:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698327.461729:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698327.461738:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698327.461741:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698327.461743:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698327.461748:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6228 00000100:00080000:1.0:1787698327.461751:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698327.461752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461754:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461756:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698327.461759:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698327.461770:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461776:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698327.461778:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698327.461783:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698327.461788:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698327.461797:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698327.461812:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698327.461819:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698327.461823:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698327.461824:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698327.461829:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6228 00000100:00080000:1.0:1787698327.461833:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698327.461836:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461839:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461841:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698327.461844:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698327.461846:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461848:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461851:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698327.461853:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698327.461858:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698327.461863:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698327.461873:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698327.461881:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698327.461888:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698327.461891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698327.461893:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698327.461896:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6228 00000100:00080000:1.0:1787698327.461900:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698327.461903:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461905:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461907:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698327.461910:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698327.461913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698327.461915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698327.461917:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698327.461919:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698327.461924:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698327.461929:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698327.461937:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698327.461946:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698327.461955:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6238) 00000100:00000040:0.0:1787698327.462048:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462063:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126324736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698327.462116:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462124:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698327.462185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698327.462205:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698327/real 1787698327] req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462271:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698327.462281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698327.462286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698327.462290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126324736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698327.462304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698327.462327:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462334:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698327.462359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698327.462367:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462378:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698327.462383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698327/real 1787698327] req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462401:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698327.462408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698327.462412:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698327.462415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698327.462453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462461:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698327.462474:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462481:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698327.462491:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698327.462498:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462514:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698327.462521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698327/real 1787698327] req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698327.462549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698327.462552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698327.462556:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462563:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126324992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698327.462569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698327.462593:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462599:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698327.462611:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698327.462619:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462628:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698327.462640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698327/real 1787698327] req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462652:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462660:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698327.462668:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698327.462671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698327.462676:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698327.462701:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698327.462723:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462730:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698327.462743:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698327.462749:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698352 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462758:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698327.462900:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698327/real 1787698327] req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462928:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698327.462936:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698327.462944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698327.462947:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698327.462951:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698327.462959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698327.462965:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126324736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698327.462976:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698327.462988:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126324864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698327.462995:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698327.463000:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126324992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698327.463006:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698327.463010:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126325120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698327.463017:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698327.463022:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126325248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698352 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698327.463037:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698327.973535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698327.973535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698327.973545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698328.101600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698328.997539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698328.997539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698328.997549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698329.125594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698329.381715:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300902767, at 4300902784 00000100:00000040:2.0:1787698329.382018:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300902767, at 4300902784 00000100:00100000:3.0:1787698330.021486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698330.021491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698330.021507:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698330.149690:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698331.045532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698331.045533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698331.045552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698331.173606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698332.069522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698332.069523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698332.069544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698332.197577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698332.581540:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698332.581551:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698332.581563:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698332.581576:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6233 00000100:00080000:1.0:1787698332.581588:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698332.581592:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581596:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581599:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698332.581604:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698332.581607:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581612:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698332.581616:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581630:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698332.581653:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698332.581664:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698332.581706:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698332.581710:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698332.581712:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698332.581717:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6233 00000100:00080000:1.0:1787698332.581720:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698332.581723:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581726:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581728:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698332.581731:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698332.581734:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581736:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581739:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698332.581741:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698332.581751:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698332.581754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698332.581756:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698332.581765:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698332.581775:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.581778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126325376:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698332.581784:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698332.581793:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698332.581795:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698332.581801:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6233 00000100:00000040:0.0:1787698332.581802:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698332.581805:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698332.581808:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698332.581811:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581813:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698332.581815:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698332.581817:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581819:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581821:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698332.581824:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698332.581828:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698332.581831:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698332.581838:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698332.581844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581847:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698332.581854:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698332.581857:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698332.581859:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698332.581860:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698332.581863:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6233 00000100:00080000:1.0:1787698332.581866:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698332.581868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581870:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581872:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698332.581874:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698332.581876:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581879:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581881:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698332.581883:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698332.581888:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698332.581893:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698332.581897:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698332/real 1787698332] req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698332.581900:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698332.581909:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.581913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698332.581916:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698332.581920:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:0.0:1787698332.581922:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698332.581928:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698332.581931:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698332.581931:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6233 00000100:00080000:0.0:1787698332.581935:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787698332.581935:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698332.581937:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698332.581939:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698332.581940:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581942:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698332.581945:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:0.0:1787698332.581948:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126325376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698332.581948:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698332.581952:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698332.581954:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698332.581959:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698332.581962:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698332.581970:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126325376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698332.581970:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787698332.581981:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698332.581981:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698332.581989:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6243) 00000100:00000040:0.0:1787698332.581996:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582004:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698332.582018:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582025:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698332.582036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698332.582042:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582049:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698332.582052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698332/real 1787698332] req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582059:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698332.582069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698332.582070:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698332.582073:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582085:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698332.582089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582095:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698332.582103:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582107:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698332.582114:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698332.582118:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582124:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698332.582127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698332/real 1787698332] req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582134:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698332.582143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698332.582145:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698332.582147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582152:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698332.582159:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698332.582172:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582176:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698332.582182:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698332.582186:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582191:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698332.582195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698332/real 1787698332] req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698332.582211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698332.582212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698332.582214:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698332.582220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698332.582241:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582246:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698332.582254:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698332.582258:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698357 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582263:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698332.582266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698332/real 1787698332] req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582272:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698332.582277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698332.582281:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698332.582283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698332.582285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698332.582288:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126325888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698332.582291:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126325504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698332.582295:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698332.582304:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126325632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698332.582308:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698332.582311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126325760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698332.582316:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698332.582319:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126325888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698357 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698332.582323:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698333.093525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698333.093525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698333.093535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698333.221591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698334.117504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698334.117504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698334.117606:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698334.245616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698334.501573:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300907887, at 4300907904 00000100:00000040:2.0:1787698334.501574:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300907887, at 4300907904 00000100:00100000:0.0:1787698335.141543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698335.141543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698335.141543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698335.269554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698336.165529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698336.165529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698336.165531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698336.293662:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698337.189534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698337.189534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698337.189534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698337.317579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698337.701555:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698337.701569:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698337.701573:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698337.701586:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6238 00000100:00080000:2.0:1787698337.701625:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698337.701632:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701639:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698337.701643:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698337.701646:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701647:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701650:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698337.701652:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701668:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698337.701694:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698337.701715:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698337.701741:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698337.701746:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698337.701747:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698337.701752:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6238 00000100:00080000:2.0:1787698337.701755:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698337.701757:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701759:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701761:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698337.701764:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698337.701766:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701769:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701771:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698337.701773:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698337.701782:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:0.0:1787698337.701786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698337.701787:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698337.701798:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698337.701803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126326016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698337.701812:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698337.701819:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698337.701821:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698337.701828:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698337.701832:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6238 00000100:00080000:2.0:1787698337.701835:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698337.701837:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698337.701839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701844:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701848:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698337.701852:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698337.701855:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701856:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701861:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698337.701862:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698337.701866:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698337.701870:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698337.701876:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698337.701882:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698337.701887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698337.701889:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698337.701890:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698337.701893:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6238 00000100:00080000:2.0:1787698337.701895:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787698337.701898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701901:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701904:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701905:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698337.701907:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698337.701909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701914:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698337.701916:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698337.701919:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698337.701921:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698337.701928:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698337.701935:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698337.701941:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698337.701945:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698337.701947:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698337.701948:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698337.701951:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6238 00000100:00080000:2.0:1787698337.701954:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698337.701956:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698337.701960:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701961:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698337.701962:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698337.701965:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698337.701967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698337/real 1787698337] req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698337.701969:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698337.701974:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698337.701980:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698337.701989:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698337.701994:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698337.701999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698337.702008:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698337.702018:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698337.702046:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6248) 00000100:00000040:0.0:1787698337.702051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698337.702060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698337.702063:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698337.702067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126326016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698337.702079:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126326016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698337.702087:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698337.702100:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698337.702116:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702120:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698337.702129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698337.702134:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702141:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698337.702145:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698337/real 1787698337] req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698337.702170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698337.702172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698337.702174:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702178:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698337.702181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698337.702193:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702198:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698337.702204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698337.702208:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702214:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698337.702217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698337/real 1787698337] req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698337.702242:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698337.702243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698337.702245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698337.702252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698337.702265:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702269:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698337.702278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698337.702284:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702293:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698337.702298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698337/real 1787698337] req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702332:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698337.702340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698337.702343:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698337.702346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698337.702356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698337.702369:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702374:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698337.702382:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698337.702386:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698362 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702392:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698337.702395:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698337/real 1787698337] req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698337.702412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698337.702416:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698337.702443:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698337.702446:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698337.702451:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698337.702455:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126326144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698337.702461:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698337.702465:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126326272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698337.702469:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698337.702473:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126326400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698337.702476:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698337.702479:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126326528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698362 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698337.702482:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698338.213539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698338.213539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698338.213611:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698338.341619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698339.237496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698339.237496:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698339.237497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698339.365632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698339.621504:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300913007, at 4300913024 00000100:00000040:3.0:1787698339.621507:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300913007, at 4300913024 00000100:00100000:0.0:1787698340.261539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698340.261539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698340.261539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698340.389620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698341.285557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698341.285557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698341.285557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698341.413632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698342.309535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698342.309535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698342.309539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698342.437603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698342.821558:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698342.821571:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698342.821576:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698342.821589:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6243 00000100:00080000:3.0:1787698342.821603:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698342.821607:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821612:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821616:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698342.821622:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698342.821625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821628:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698342.821634:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821653:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698342.821685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698342.821698:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698342.821716:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698342.821719:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698342.821721:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698342.821725:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6243 00000100:00080000:3.0:1787698342.821729:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698342.821732:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821753:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821756:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:0.0:1787698342.821758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698342.821759:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698342.821762:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821765:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821767:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698342.821769:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698342.821773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126326656:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698342.821778:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698342.821783:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698342.821793:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698342.821802:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698342.821809:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787698342.821811:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698342.821813:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698342.821814:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698342.821818:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6243 00000100:00080000:3.0:1787698342.821822:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698342.821823:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821825:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821828:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821830:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698342.821832:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698342.821834:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821837:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821838:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698342.821840:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698342.821850:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698342.821854:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698342.821862:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698342.821864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821871:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698342.821878:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698342.821881:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:0.0:1787698342.821883:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698342.821883:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698342.821887:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6243 00000100:00080000:3.0:1787698342.821891:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698342.821893:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821895:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821897:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698342.821899:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698342.821902:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698342.821903:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821905:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698342.821907:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698342.821912:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698342.821915:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698342.821918:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698342/real 1787698342] req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698342.821923:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698342.821932:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.821944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698342.821950:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698342.821953:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:0.0:1787698342.821954:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698342.821955:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698342.821959:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6243 00000100:00080000:3.0:1787698342.821962:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787698342.821963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698342.821965:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698342.821967:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698342.821968:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821970:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698342.821972:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698342.821975:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698342.821977:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698342.821978:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698342.821980:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698342.821983:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787698342.821986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126326656:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698342.821988:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698342.821992:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698342.821993:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126326656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698342.822000:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787698342.822003:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698342.822007:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698342.822017:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6253) 00000100:00000040:0.0:1787698342.822031:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822041:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698342.822055:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822060:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698342.822070:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698342.822076:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822086:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698342.822091:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698342/real 1787698342] req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822102:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698342.822124:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698342.822128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698342.822131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698342.822139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822155:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698342.822165:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822169:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698342.822177:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698342.822182:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822188:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698342.822192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698342/real 1787698342] req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698342.822219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698342.822222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698342.822226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126326912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698342.822240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698342.822273:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822278:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698342.822286:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698342.822291:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822297:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698342.822301:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698342/real 1787698342] req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822313:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698342.822318:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698342.822319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698342.822321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698342.822330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698342.822340:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822355:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698342.822367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698342.822374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698367 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822383:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698342.822388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698342/real 1787698342] req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698342.822404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698342.822409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698342.822410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698342.822412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698342.822449:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698342.822457:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126326784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698342.822465:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698342.822471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126326912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698342.822480:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698342.822485:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126327040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698342.822498:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698342.822517:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126327168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698367 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698342.822522:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698343.333541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698343.333542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698343.333542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698343.461673:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698344.357538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698344.357538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698344.357632:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698344.485631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698344.741517:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300918127, at 4300918144 00000100:00000040:3.0:1787698344.741517:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300918127, at 4300918144 00000100:00100000:0.0:1787698345.381500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698345.381501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698345.381518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698345.509628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698346.405553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698346.405553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698346.406319:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698346.533629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698347.429577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698347.429577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698347.429578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698347.557647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698347.942011:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698347.942023:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698347.942028:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698347.942040:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6248 00000100:00080000:2.0:1787698347.942052:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698347.942056:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942061:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942065:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698347.942070:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698347.942073:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942075:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942078:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698347.942082:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942100:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698347.942156:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698347.942174:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698347.942198:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698347.942201:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698347.942203:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698347.942207:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6248 00000100:00080000:2.0:1787698347.942211:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698347.942214:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942216:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942217:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698347.942220:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698347.942222:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942225:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942227:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698347.942229:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698347.942239:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698347.942243:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698347.942253:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698347.942259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698347.942264:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942281:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126327296:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698347.942303:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698347.942307:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698347.942309:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698347.942313:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6248 00000100:00080000:2.0:1787698347.942317:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698347.942319:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698347.942320:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942326:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942330:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698347.942334:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698347.942337:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698347.942340:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942341:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942344:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698347.942346:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698347.942350:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698347.942360:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698347.942370:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698347.942380:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698347.942389:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698347.942394:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698347.942399:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698347.942402:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698347.942406:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6248 00000100:00080000:2.0:1787698347.942409:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698347.942411:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698347.942453:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698347.942456:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942462:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698347.942466:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698347.942472:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942475:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942478:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698347.942481:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698347.942487:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698347.942491:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698347.942496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698347/real 1787698347] req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698347.942501:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698347.942517:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698347.942523:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698347.942525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698347.942526:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698347.942529:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698347.942538:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6248 00000100:00000040:1.0:1787698347.942542:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698347.942543:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698347.942550:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942557:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942563:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698347.942567:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:1.0:1787698347.942571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698347.942572:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698347.942578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698347.942579:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698347.942584:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698347.942586:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698347.942591:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698347.942600:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698347.942603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126327296:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787698347.942606:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698347.942613:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126327296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698347.942620:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787698347.942626:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698347.942633:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698347.942644:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6259) 00000100:00000040:1.0:1787698347.942648:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942664:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698347.942683:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942690:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698347.942701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698347.942707:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942717:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698347.942723:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698347/real 1787698347] req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942749:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698347.942756:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698347.942759:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698347.942763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698347.942775:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698347.942795:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942802:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698347.942813:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698347.942820:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942828:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698347.942834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698347/real 1787698347] req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942854:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698347.942862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698347.942866:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698347.942869:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698347.942882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942891:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698347.942902:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942910:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698347.942933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698347.942939:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942949:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698347.942954:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698347/real 1787698347] req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.942971:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698347.942979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698347.942982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698347.942986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.942994:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698347.942999:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.943008:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698347.943018:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.943025:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698347.943033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698347.943039:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698372 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.943047:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698347.943052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698347/real 1787698347] req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.943063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698347.943077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698347.943084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698347.943087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698347.943091:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698347.943098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126327808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698347.943103:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126327424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698347.943114:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698347.943120:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126327552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698347.943126:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698347.943131:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126327680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698347.943137:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698347.943141:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126327808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698372 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698347.943147:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698348.453539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698348.453539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698348.453539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698348.582341:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698349.477524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698349.477524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698349.477524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698349.605607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698349.861546:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300923248, at 4300923264 00000100:00000040:2.0:1787698349.861573:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300923248, at 4300923264 00000100:00100000:0.0:1787698350.501532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698350.501532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698350.501602:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698350.629603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698351.525582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698351.525583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698351.527120:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698351.653641:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698352.549547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698352.549553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698352.549575:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698352.677609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698353.061567:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698353.061577:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698353.061582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698353.061596:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6254 00000100:00080000:3.0:1787698353.061610:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698353.061614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.061619:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061623:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698353.061630:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698353.061633:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.061636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061640:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698353.061643:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061662:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698353.061696:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698353.061714:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698353.061737:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698353.061739:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698353.061741:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698353.061744:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6254 00000100:00080000:3.0:1787698353.061747:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698353.061750:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.061772:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061775:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698353.061778:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698353.061785:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698353.061787:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698353.061789:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061794:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698353.061801:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698353.061814:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126327936:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698353.061816:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698353.061826:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698353.061841:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698353.061857:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698353.061857:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.061881:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698353.061882:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698353.061886:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698353.061887:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698353.061891:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6254 00000100:00080000:3.0:1787698353.061895:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698353.061899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.061902:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061904:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698353.061907:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698353.061910:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.061913:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061915:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698353.061918:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:1.0:1787698353.061929:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698353.061933:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698353.061939:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698353.061948:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698353.061954:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698353.061962:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698353.061974:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698353.061977:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698353.061978:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698353.061982:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6254 00000100:00080000:3.0:1787698353.061985:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698353.061988:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.061992:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.061994:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:1.0:1787698353.061998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698353.061999:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698353.062007:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.062015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698353.062027:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698353.062027:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698353.062035:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698353.062042:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698353.062046:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698353.062048:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698353.062060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787698353.062061:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698353.062068:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698353.062077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698353.062087:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126327936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698353.062105:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126327936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698353.062124:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698353.062126:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698353.062131:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698353.062133:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698353.062138:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6254 00000100:00080000:3.0:1787698353.062142:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698353.062146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698353.062147:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698353.062153:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.062157:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698353.062161:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698353.062166:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698353.062169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698353.062173:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698353.062180:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698353.062185:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698353.062191:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698353.062194:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698353.062198:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698353.062207:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698353.062213:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698353.062222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698353.062224:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062233:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698353.062236:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6264) 00000100:00000040:1.0:1787698353.062248:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698353.062255:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062276:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698353.062284:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698353.062286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698353.062290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698353.062303:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698353.062342:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062350:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698353.062361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698353.062367:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062375:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698353.062380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062398:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698353.062405:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698353.062444:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698353.062448:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698353.062460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698353.062485:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062491:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698353.062501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698353.062507:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062515:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698353.062519:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698353.062542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698353.062544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698353.062547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698353.062557:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126328064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698353.062563:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698353.062568:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126328192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698353.062574:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698353.062942:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126328320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698353.062949:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698353.062957:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.062964:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698353.062977:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.062983:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698353.062993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698353.062999:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698378 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.063007:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698353.063012:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698353/real 1787698353] req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.063021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698353.063029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698353.063036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698353.063038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698353.063041:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698353.063048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698353.063053:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126328448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698378 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698353.063064:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698353.573529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698353.573538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698353.573802:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698353.701638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698354.597513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698354.597524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698354.597528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698354.725565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698354.981541:0:88404:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300928367, at 4300928384 00000100:00000040:2.0:1787698354.981572:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300928367, at 4300928384 00000100:00100000:1.0:1787698355.621517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698355.621528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698355.621645:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698355.749606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698356.645506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698356.645506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698356.645698:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698356.773552:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698357.669545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698357.669570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698357.669576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698357.797647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698358.181546:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698358.181559:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698358.181584:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698358.181597:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6259 00000100:00080000:3.0:1787698358.181611:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698358.181615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.181620:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.181624:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698358.181629:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698358.181632:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.181635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.181638:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698358.181642:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698358.181656:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698358.181685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698358.181715:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698358.181742:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698358.181746:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698358.181748:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698358.181752:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6259 00000100:00080000:3.0:1787698358.181756:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698358.181759:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.181761:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.181764:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698358.181767:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698358.181770:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.181773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.181781:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698358.181785:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698358.181796:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698358.181803:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698358.181804:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698358.181817:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698358.181838:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.181841:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126328576:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698358.181855:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698358.181858:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698358.181860:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698358.181865:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6259 00000100:00080000:3.0:1787698358.181869:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698358.181873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.181876:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.181878:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698358.181880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698358.181886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698358.181888:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698358.181891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.181896:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698358.181903:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698358.181907:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698358.181911:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698358.181917:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698358.181930:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698358.181952:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698358.181958:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698358.181967:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698358.181971:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698358.181973:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698358.181979:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6259 00000100:00000040:1.0:1787698358.181983:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698358.181985:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698358.181991:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.181997:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.182000:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698358.182003:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698358.182006:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.182008:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698358.182011:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698358.182014:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698358.182021:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698358.182025:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698358/real 1787698358] req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698358.182031:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698358.182052:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698358.182053:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698358.182079:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182083:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698358.182100:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698358.182104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698358.182105:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698358.182110:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698358.182112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698358.182117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698358.182127:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6259 00000100:00100000:1.0:1787698358.182133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126328576:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698358.182133:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698358.182140:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.182145:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698358.182146:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126328576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698358.182149:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698358.182153:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698358.182159:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698358.182162:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698358.182163:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:3.0:1787698358.182165:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698358.182177:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698358.182183:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698358.182183:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698358.182192:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698358.182198:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328704:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787698358.182206:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698358.182215:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698358.182221:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182236:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698358.182261:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6269) 00000800:00000100:1.0:1787698358.182266:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698358.182275:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182287:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698358.182300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698358/real 1787698358] req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698358.182336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698358.182339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698358.182343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698358.182357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698358.182376:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182391:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698358.182401:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698358.182407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182416:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698358.182452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698358/real 1787698358] req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182473:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698358.182481:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698358.182491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698358.182495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698358.182508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698358.182528:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182536:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698358.182551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698358.182558:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182568:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698358.182573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698358/real 1787698358] req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182584:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182593:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698358.182600:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698358.182603:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698358.182607:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126328960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698358.182621:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126328704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698358.182629:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698358.182634:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126328832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698358.182641:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698358.182646:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126328960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698358.182653:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698358.182660:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698358.182685:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182691:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698358.182701:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698358.182707:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698383 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182716:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698358.182720:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698358/real 1787698358] req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698358.182746:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698358.182753:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698358.182756:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698358.182760:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698358.182768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698358.182772:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126329088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698383 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698358.182778:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698358.693498:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698358.693524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698358.693524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698358.821645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698359.717545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698359.717545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698359.717563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698359.845538:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698360.101522:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300933487, at 4300933504 00000100:00000040:2.0:1787698360.101533:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300933487, at 4300933504 00000100:00100000:0.0:1787698360.741553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698360.741590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698360.741597:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698360.869607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698361.765553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698361.765569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698361.765597:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698361.893634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698362.789565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698362.789568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698362.789586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698362.917626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698363.301581:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698363.301595:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698363.301600:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698363.301612:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6264 00000100:00080000:1.0:1787698363.301633:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698363.301638:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301645:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301650:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698363.301658:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698363.301662:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301666:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301670:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698363.301674:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301691:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698363.301724:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698363.301750:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698363.301769:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698363.301771:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698363.301773:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698363.301776:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6264 00000100:00080000:1.0:1787698363.301779:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698363.301781:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301782:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301784:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698363.301786:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698363.301789:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301790:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301792:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698363.301795:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698363.301805:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698363.301809:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698363.301817:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698363.301822:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698363.301843:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.301847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126329216:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698363.301860:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698363.301863:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698363.301865:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698363.301870:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6264 00000100:00080000:1.0:1787698363.301874:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698363.301877:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698363.301882:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698363.301883:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301889:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698363.301893:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698363.301896:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301898:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301901:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301903:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698363.301905:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698363.301909:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698363.301913:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698363.301920:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698363.301927:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698363.301932:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698363.301935:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787698363.301938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698363.301938:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698363.301945:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6264 00000100:00080000:1.0:1787698363.301949:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698363.301951:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301954:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698363.301958:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698363.301964:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698363.301967:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698363.301972:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.301974:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.301975:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698363.301977:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698363.301981:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698363.301984:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698363.301992:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698363.301999:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698363/real 1787698363] req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698363.302008:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698363.302013:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698363.302015:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698363.302020:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6264 00000100:00080000:1.0:1787698363.302022:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698363.302025:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.302028:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.302030:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698363.302032:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698363.302036:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698363.302038:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698363.302040:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698363.302045:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698363.302051:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698363.302054:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698363.302063:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698363.302073:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698363.302094:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6274) 00000100:00100000:0.0:1787698363.302138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302173:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698363.302180:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698363.302183:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698363.302186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302192:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126329216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698363.302197:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126329216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698363.302229:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698363.302244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302251:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698363.302262:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302267:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698363.302277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698363.302281:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302290:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698363.302297:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698363/real 1787698363] req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302308:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698363.302323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698363.302326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698363.302330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302337:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698363.302351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698363.302373:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302380:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698363.302391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698363.302407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302417:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698363.302444:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698363/real 1787698363] req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698363.302463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698363.302465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698363.302467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698363.302475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302486:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698363.302505:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302513:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698363.302525:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698363.302532:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302542:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698363.302552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698363/real 1787698363] req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302573:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698363.302580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698363.302583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698363.302585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698363.302600:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302607:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698363.302626:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302633:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698363.302644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698363.302651:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698388 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302659:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698363.302665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698363/real 1787698363] req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302676:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698363.302685:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698363.302701:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698363.302703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698363.302706:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698363.302710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698363.302714:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126329344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698363.302719:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698363.302724:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126329472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698363.302745:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698363.302751:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126329600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698363.302758:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698363.302763:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126329728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698388 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698363.302770:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698363.813544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698363.813544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698363.813544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698363.941627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698364.837547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698364.837547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698364.837562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698364.965626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698365.221562:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300938607, at 4300938624 00000100:00000040:3.0:1787698365.221570:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300938607, at 4300938624 00000100:00100000:0.0:1787698365.861548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698365.861548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698365.861560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698365.989650:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698366.885536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698366.885536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698366.885538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698367.013657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698367.909545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698367.909545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698367.909657:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698368.037569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698368.421566:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698368.421590:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698368.421594:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698368.421603:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6269 00000100:00080000:3.0:1787698368.421615:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698368.421618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421621:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.421623:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698368.421627:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698368.421629:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421631:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.421639:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698368.421642:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698368.421655:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698368.421679:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698368.421691:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698368.421708:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698368.421710:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698368.421712:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698368.421715:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6269 00000100:00080000:3.0:1787698368.421718:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698368.421719:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421721:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.421723:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698368.421725:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698368.421732:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421734:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.421736:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698368.421738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698368.421745:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698368.421748:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698368.421761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698368.421764:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698368.421814:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.421818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126329856:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698368.421838:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698368.421841:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698368.421844:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698368.421849:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6269 00000100:00080000:3.0:1787698368.421853:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698368.421858:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421867:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698368.421868:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698368.421872:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698368.421878:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698368.421886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698368.421892:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421898:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.421902:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698368.421905:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698368.421912:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698368.421918:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698368.421927:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787698368.421940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421946:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698368.421954:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698368.421957:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698368.421960:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698368.421964:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698368.421965:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6269 00000100:00080000:3.0:1787698368.421978:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698368.421982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.421986:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698368.421987:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698368.421989:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.421999:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.422010:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698368.422020:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698368.422025:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698368.422029:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698368.422036:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698368.422042:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698368.422048:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787698368.422051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698368/real 1787698368] req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698368.422052:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698368.422056:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698368.422081:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6269 00000100:00080000:3.0:1787698368.422085:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698368.422088:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698368.422091:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.422093:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698368.422095:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698368.422108:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698368.422116:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698368.422117:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698368.422128:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698368.422137:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698368.422139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698368.422146:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787698368.422157:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787698368.422157:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787698368.422168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:3.0:1787698368.422175:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698368.422177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126329856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698368.422245:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126329856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698368.422246:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698368.422262:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698368.422270:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6279) 00000100:00000040:0.0:1787698368.422282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698368.422303:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422308:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698368.422319:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698368.422324:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422341:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698368.422345:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698368/real 1787698368] req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422352:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698368.422362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698368.422364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698368.422366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126329984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698368.422375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698368.422390:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422397:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698368.422408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698368.422415:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422454:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698368.422466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698368/real 1787698368] req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698368.422495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698368.422497:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698368.422499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698368.422510:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422518:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698368.422533:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422540:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698368.422553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698368.422560:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422570:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698368.422575:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698368/real 1787698368] req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422608:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698368.422615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698368.422619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698368.422622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698368.422630:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698368.422645:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422650:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698368.422658:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698368.422663:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698393 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422669:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698368.422672:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698368/real 1787698368] req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422678:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698368.422689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698368.422696:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698368.422699:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698368.422703:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698368.422710:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698368.422715:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126329984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698368.422724:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698368.422730:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126330112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698368.422738:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698368.422743:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126330240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698368.422748:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698368.422750:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126330368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698393 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698368.422755:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698368.933529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698368.933551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698368.933570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698369.061603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698369.957548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698369.957548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698369.957549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698370.085599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698370.341535:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300943727, at 4300943744 00000100:00000040:2.0:1787698370.341559:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300943727, at 4300943744 00000100:00100000:0.0:1787698370.981523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698370.981523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698370.981523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698371.109595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698372.005511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698372.005511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698372.005517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698372.133597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698373.029549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698373.029549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698373.029550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698373.157552:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698373.541576:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698373.541586:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698373.541591:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698373.541601:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6274 00000100:00080000:1.0:1787698373.541611:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698373.541614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.541618:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698373.541622:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698373.541627:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698373.541630:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.541633:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698373.541636:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698373.541647:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698373.541662:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698373.541685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698373.541699:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698373.541719:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698373.541723:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698373.541725:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698373.541729:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6274 00000100:00080000:1.0:1787698373.541733:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698373.541737:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.541741:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698373.541743:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698373.541747:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698373.541748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698373.541754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.541760:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698373.541763:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00100000:0.0:1787698373.541764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126330496:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698373.541769:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698373.541779:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698373.541785:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698373.541796:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698373.541798:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698373.541811:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698373.541820:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698373.541830:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698373.541834:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698373.541836:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698373.541840:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6274 00000800:00000100:0.0:1787698373.541843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698373.541847:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698373.541852:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.541856:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698373.541860:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698373.541860:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698373.541865:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698373.541871:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.541875:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698373.541877:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698373.541880:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698373.541885:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787698373.541889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698373/real 1787698373] req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698373.541892:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698373.541906:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698373.541907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.541934:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698373.541936:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698373.541956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698373.541957:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698373.541964:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698373.541965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787698373.541969:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698373.541972:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698373.541990:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6274 00000100:00080000:1.0:1787698373.541995:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698373.541999:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.542005:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698373.542008:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126330496:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698373.542009:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698373.542012:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698373.542016:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698373.542018:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698373.542021:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698373.542026:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698373.542031:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126330496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698373.542033:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698373.542041:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:1.0:1787698373.542042:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698373.542054:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698373.542058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542077:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698373.542082:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542092:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542098:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698373.542103:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698373.542107:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787698373.542110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698373.542111:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787698373.542117:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698373.542119:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6274 00000100:00080000:1.0:1787698373.542127:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698373.542130:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787698373.542132:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787698373.542136:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698373/real 1787698373] req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698373.542139:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698373.542145:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698373.542150:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:0.0:1787698373.542152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698373.542163:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698373.542164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698373.542172:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:0.0:1787698373.542173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787698373.542178:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698373.542179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787698373.542186:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787698373.542187:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698373.542195:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698373.542202:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698373.542207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698373.542218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698373.542218:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787698373.542231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698373.542242:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542246:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542267:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698373.542273:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6284) 00000800:00000100:0.0:1787698373.542278:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698373.542283:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542290:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698373.542294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698373/real 1787698373] req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542301:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698373.542311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698373.542313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698373.542315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698373.542326:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126330624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698373.542331:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698373.542334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126330752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698373.542338:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698373.542343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698373.542356:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542360:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698373.542367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698373.542370:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542376:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698373.542379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698373/real 1787698373] req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698373.542395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698373.542400:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698373.542403:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126330880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698373.542410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698373.542452:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542460:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698373.542470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698373.542477:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698398 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542486:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698373.542491:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698373/real 1787698373] req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542503:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698373.542508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698373.542513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698373.542514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698373.542516:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698373.542525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698373.542529:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126330880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698373.542534:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698373.542538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126331008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698398 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698373.542543:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698374.053507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698374.053557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698374.053575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698374.181585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698375.077507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698375.077514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698375.077529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698375.205586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698375.461514:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300948847, at 4300948864 00000100:00000040:3.0:1787698375.461514:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300948847, at 4300948864 00000100:00100000:0.0:1787698376.101512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698376.101512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698376.101512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698376.229651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698377.125583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698377.125583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698377.125590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698377.253635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698378.149483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698378.149528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698378.149543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698378.277575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698378.661574:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698378.661584:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698378.661589:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698378.661599:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6279 00000100:00080000:2.0:1787698378.661611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698378.661614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661621:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698378.661626:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698378.661629:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661631:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661634:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698378.661636:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661651:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698378.661679:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698378.661692:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698378.661710:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698378.661713:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698378.661715:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698378.661718:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6279 00000100:00080000:2.0:1787698378.661721:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698378.661722:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661724:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661725:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698378.661727:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698378.661730:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661731:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661733:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698378.661735:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698378.661743:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698378.661747:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698378.661753:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698378.661761:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698378.661767:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698378.661769:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698378.661770:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698378.661773:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6279 00000100:00080000:2.0:1787698378.661775:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698378.661777:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661779:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661781:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698378.661784:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698378.661785:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698378.661788:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661793:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661796:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698378.661798:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698378.661802:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698378.661805:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698378.661813:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698378.661825:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.661826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126331136:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698378.661840:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698378.661843:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698378.661844:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698378.661847:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6279 00000100:00080000:2.0:1787698378.661848:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698378.661850:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661851:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661853:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698378.661854:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698378.661856:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661858:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661859:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698378.661861:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698378.661865:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698378.661870:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698378.661874:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698378.661879:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698378.661908:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661918:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698378.661925:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698378.661927:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698378.661929:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698378.661932:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6279 00000100:00080000:2.0:1787698378.661934:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698378.661936:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661938:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661940:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698378.661942:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698378.661944:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698378.661946:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698378.661948:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698378.661950:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698378.661953:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698378.661956:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:0.0:1787698378.661962:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698378.661965:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698378.661973:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698378.661982:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6289) 00000100:00000040:0.0:1787698378.661987:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698378/real 1787698378] req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662087:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698378.662108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698378.662112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698378.662117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126331136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698378.662135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698378.662159:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662168:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698378.662181:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698378.662189:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662199:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698378.662203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698378/real 1787698378] req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662223:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698378.662229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698378.662232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698378.662234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698378.662243:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126331136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698378.662252:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698378.662258:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126331264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698378.662263:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698378.662274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698378.662297:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662306:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698378.662318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698378.662326:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662349:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787698378.662354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698378/real 1787698378] req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662362:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698378.662381:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698378.662384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698378.662389:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698378.662405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698378.662460:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662466:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698378.662475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698378.662480:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662487:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787698378.662496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698378/real 1787698378] req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662510:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698378.662515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698378.662517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698378.662519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662524:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698378.662528:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662534:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698378.662542:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662547:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787698378.662554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698378.662584:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698403 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662592:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698378.662596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698378/real 1787698378] req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698378.662620:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698378.662625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698378.662627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698378.662629:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698378.662634:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698378.662638:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126331392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698378.662644:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698378.662648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126331520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698378.662653:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787698378.662656:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126331648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698403 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698378.662661:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698379.173540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698379.173540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698379.173558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698379.301586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698380.197561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698380.197561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698380.197561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698380.325647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698380.581559:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300953967, at 4300953984 00000100:00000040:3.0:1787698380.581560:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300953967, at 4300953984 00000100:00100000:1.0:1787698381.221547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698381.221547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698381.221552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698381.349580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698382.245540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698382.245540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698382.245540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698382.373628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698383.269561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698383.269561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698383.269561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698383.397605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698383.781601:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698383.781615:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698383.781621:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698383.781633:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6284 00000100:00080000:2.0:1787698383.781646:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698383.781650:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.781656:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.781660:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698383.781666:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698383.781670:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.781673:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.781677:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698383.781681:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698383.781701:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698383.781739:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698383.781752:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698383.781779:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698383.781783:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698383.781784:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698383.781788:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6284 00000100:00080000:2.0:1787698383.781792:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698383.781794:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.781797:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.781799:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698383.781817:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698383.781819:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.781822:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.781824:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698383.781826:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698383.781836:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698383.781842:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698383.781843:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698383.781854:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698383.781861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126331776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698383.781870:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698383.781879:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698383.781887:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698383.781888:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698383.781891:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6284 00000100:00080000:2.0:1787698383.781895:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698383.781897:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698383.781901:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698383.781903:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.781908:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698383.781912:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698383.781919:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698383.781920:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.781926:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.781929:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698383.781932:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698383.781938:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698383.781944:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698383.781955:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698383.781964:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698383.781974:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698383.781981:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698383.781986:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698383.781986:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698383.781989:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698383.781997:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6284 00000100:00080000:2.0:1787698383.782003:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698383.782007:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.782010:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.782012:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698383.782014:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698383.782016:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.782018:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.782019:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698383.782021:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698383.782026:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698383.782030:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698383.782038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698383/real 1787698383] req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698383.782039:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698383.782072:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698383.782079:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698383.782083:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787698383.782084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698383.782085:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698383.782091:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6284 00000100:00080000:2.0:1787698383.782096:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698383.782101:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698383.782104:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698383.782109:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698383.782114:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698383.782119:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698383.782126:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698383.782130:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698383.782130:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698383.782136:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698383.782141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:2.0:1787698383.782141:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698383.782150:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698383.782154:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698383.782163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126331776:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787698383.782165:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698383.782172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126331776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698383.782181:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787698383.782183:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698383.782197:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782204:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698383.782215:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6294) 00000100:00100000:1.0:1787698383.782216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698383.782231:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782237:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698383.782259:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698383.782265:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782273:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698383.782278:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698383/real 1787698383] req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698383.782297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698383.782299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698383.782301:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782306:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126331904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698383.782309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698383.782324:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782329:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698383.782335:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698383.782339:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782352:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698383.782356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698383/real 1787698383] req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698383.782376:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698383.782378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698383.782380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698383.782388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698383.782402:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782408:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698383.782414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698383.782453:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782461:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698383.782473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698383/real 1787698383] req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698383.782493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698383.782494:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698383.782497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698383.782511:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126331904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698383.782519:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698383.782526:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126332032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698383.782534:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698383.782540:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126332160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698383.782549:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698383.782558:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782565:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698383.782575:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782581:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698383.782590:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698383.782597:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698408 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782607:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698383.782613:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698383/real 1787698383] req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782626:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698383.782635:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698383.782644:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698383.782648:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698383.782651:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698383.782657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698383.782663:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126332288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698408 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698383.782669:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698384.293531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698384.293531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698384.293531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698384.421613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698385.317562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698385.317562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698385.317562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698385.445700:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698385.701499:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300959087, at 4300959104 00000100:00000040:2.0:1787698385.701506:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300959087, at 4300959104 00000100:00100000:3.0:1787698386.341552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698386.341553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698386.341600:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698386.469522:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698387.365571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698387.365577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698387.365599:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698387.493580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698388.389545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698388.389554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698388.389592:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698388.517606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698388.901566:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698388.901576:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698388.901580:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698388.901592:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6289 00000100:00080000:1.0:1787698388.901604:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698388.901607:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901613:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698388.901618:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698388.901620:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901622:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901625:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698388.901627:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901647:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698388.901675:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698388.901693:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698388.901735:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698388.901738:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698388.901740:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698388.901744:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6289 00000100:00080000:1.0:1787698388.901748:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698388.901750:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901752:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901754:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698388.901756:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698388.901759:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901760:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901762:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698388.901764:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698388.901773:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698388.901777:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698388.901780:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698388.901787:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698388.901798:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698388.901800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126332416:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698388.901812:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698388.901815:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698388.901816:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698388.901819:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6289 00000100:00080000:1.0:1787698388.901821:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698388.901822:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901824:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901825:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698388.901827:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698388.901829:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901830:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901831:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698388.901833:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698388.901836:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698388.901839:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698388.901842:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698388.901847:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698388.901869:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901871:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698388.901878:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698388.901879:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698388.901881:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698388.901883:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6289 00000100:00080000:1.0:1787698388.901899:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698388.901902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901903:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901904:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698388.901905:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698388.901907:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901909:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901911:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698388.901913:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698388.901916:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698388.901921:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698388.901926:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698388.901932:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698388.901938:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698388.901941:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698388.901942:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698388.901945:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6289 00000100:00080000:1.0:1787698388.901947:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698388.901948:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901951:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901953:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:0.0:1787698388.901955:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698388.901955:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698388.901960:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698388.901961:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698388.901962:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698388.901966:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698388.901972:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698388.901976:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698388.901977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332544:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698388.901984:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698388.901992:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698388.901995:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698388.902013:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698388.902014:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6300) 00000100:00000040:0.0:1787698388.902027:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:0.0:1787698388.902033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698388.902040:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698388.902049:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698388.902054:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698388.902061:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:0.0:1787698388.902080:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698388.902092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698388.902102:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698388.902107:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698388.902116:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:0.0:1787698388.902122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698388.902127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698388.902137:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698388.902144:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698388.902163:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698389.413530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698389.413531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698389.413533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698389.541543:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698390.437523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698390.437523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698390.437535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698390.565615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698390.821501:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300964207, at 4300964224 00000100:00000040:2.0:1787698390.821511:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300964207, at 4300964224 00000100:00100000:0.0:1787698391.461548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698391.461548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698391.461548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698391.589597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787698391.973850:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787698391.973911:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787698391.973930:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698391.973985:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698391.974017:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698391.974032:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698391.974072:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698391.974081:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698391.974092:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698391.974100:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698391.974109:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698391.974117:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698413 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974220:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698388/real 1787698391] req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974256:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698391.974267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698391.974278:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698391.974286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698391.974289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332928:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698391.974303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698388/real 1787698391] req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698391.974329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698391.974337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698391.974340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698391.974344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974351:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332544:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698391.974357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698388/real 1787698391] req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698391.974375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698391.974382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698391.974386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698391.974388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332672:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698391.974401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698388/real 1787698391] req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698391.974459:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698391.974467:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698391.974470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698391.974474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126332800:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698391.974487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698388/real 1787698391] req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974499:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698391.974508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698391.974516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698391.974519:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698391.974523:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698391.974533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126332416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698391.974538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126332928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698391.974555:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698391.974565:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126332544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698391.974572:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698391.974577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126332672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698391.974585:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698391.974590:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126332800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698391.974597:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698391.974602:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126332416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698413 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698391.974609:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698392.485548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698392.485548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698392.485553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698392.613639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698393.509522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698393.509522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698393.509522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698393.637600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698394.021540:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698394.021553:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698394.021557:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698394.021568:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6295 00000100:00080000:1.0:1787698394.021578:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698394.021581:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021585:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021587:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698394.021592:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698394.021595:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021597:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021600:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698394.021603:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021613:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698394.021658:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698394.021670:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698394.021698:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698394.021701:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698394.021703:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698394.021707:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6295 00000100:00080000:1.0:1787698394.021711:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698394.021713:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021716:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021718:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698394.021721:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698394.021723:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021727:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698394.021730:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698394.021740:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698394.021746:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698394.021756:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698394.021767:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698394.021774:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698394.021777:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698394.021778:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698394.021781:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6295 00000100:00080000:1.0:1787698394.021784:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698394.021785:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021787:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021789:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698394.021791:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698394.021793:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021795:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021798:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698394.021800:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698394.021806:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698394.021811:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698394.021820:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698394.021831:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.021841:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698394.021843:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698394.021857:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698394.021860:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698394.021868:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6295 00000100:00100000:2.0:1787698394.021868:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126333056:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698394.021874:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698394.021878:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021881:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021883:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698394.021886:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698394.021888:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021894:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698394.021896:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698394.021902:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698394.021906:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698394.021910:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698394.021919:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698394.021925:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021936:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698394.021944:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698394.021947:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698394.021949:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698394.021953:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6295 00000100:00080000:1.0:1787698394.021957:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698394.021959:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021962:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021964:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698394.021966:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698394.021969:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698394.021973:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698394.021975:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698394.021977:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698394.021983:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698394.021987:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:2.0:1787698394.021989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698394.021999:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698394.022009:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022013:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698394.022022:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6305) 00000100:00100000:2.0:1787698394.022059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022078:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698394.022118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698394.022122:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698394.022127:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022135:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126333056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698394.022142:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022151:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698394.022167:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022175:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698394.022188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698394.022195:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022206:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698394.022212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698394.022240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698394.022244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698394.022247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698394.022260:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126333056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698394.022270:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698394.022280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126333184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698394.022287:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698394.022302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698394.022322:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022329:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698394.022339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698394.022346:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022354:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698394.022359:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698394.022387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698394.022390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698394.022393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022400:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698394.022405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698394.022454:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022463:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698394.022473:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698394.022480:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022488:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698394.022494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022512:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698394.022519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698394.022522:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698394.022525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698394.022537:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698394.022557:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022563:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698394.022572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698394.022578:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698419 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022586:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698394.022590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698394/real 1787698394] req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698394.022606:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698394.022612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698394.022617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698394.022621:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698394.022642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698394.022648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126333312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698394.022656:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698394.022663:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126333440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698394.022669:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698394.022672:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126333568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698419 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698394.022679:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698394.533532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698394.533532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698394.533533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698394.661599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698395.557499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698395.557499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698395.557543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698395.685576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698395.941517:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300969327, at 4300969344 00000100:00000040:2.0:1787698395.941520:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300969327, at 4300969344 00000100:00100000:3.0:1787698396.581556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698396.581572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698396.581572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698396.709617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698397.605499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698397.605516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698397.605529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698397.733652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698398.629599:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698398.629599:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698398.629600:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698398.757664:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698399.141555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698399.141568:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698399.141574:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698399.141590:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6300 00000100:00080000:1.0:1787698399.141602:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698399.141605:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141609:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141612:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698399.141617:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698399.141620:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141621:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141624:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698399.141626:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141642:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698399.141668:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698399.141680:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698399.141700:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698399.141702:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698399.141704:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698399.141707:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6300 00000100:00080000:1.0:1787698399.141709:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698399.141711:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141712:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141713:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698399.141716:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698399.141718:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141720:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141722:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698399.141724:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698399.141733:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698399.141736:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698399.141750:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698399.141752:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698399.141773:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698399.141815:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698399.141818:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698399.141819:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698399.141822:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6300 00000100:00100000:2.0:1787698399.141826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126333696:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698399.141827:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698399.141837:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141841:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141842:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698399.141844:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698399.141845:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141846:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141848:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698399.141849:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698399.141853:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698399.141857:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698399.141864:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698399.141870:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698399.141876:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698399.141880:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:2.0:1787698399.141881:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698399.141883:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698399.141890:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6300 00000100:00080000:1.0:1787698399.141896:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698399.141903:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141909:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141917:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698399.141919:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698399.141924:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141929:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141933:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.141934:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698399.141936:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698399.141941:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698399.141944:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698399.141949:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698399.141955:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698399.141966:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698399.141966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698399.141973:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698399.141975:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698399.141978:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6300 00000100:00080000:1.0:1787698399.141982:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698399.141985:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.141991:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698399.141992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698399.142003:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698399.142010:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698399.142018:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698399.142022:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698399.142025:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698399.142028:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698399.142036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698399.142039:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698399.142046:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698399.142069:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698399.142072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698399.142085:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698399.142121:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6310) 00000100:00000040:2.0:1787698399.142126:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698399.142139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698399.142144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698399.142149:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126333696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698399.142162:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126333696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698399.142170:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698399.142184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698399.142215:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142220:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698399.142231:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698399.142236:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142243:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698399.142246:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142254:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698399.142264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698399.142271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698399.142273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142280:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698399.142285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698399.142313:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142322:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698399.142333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698399.142340:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142348:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698399.142354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698399.142387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698399.142390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698399.142395:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142402:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126333952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698399.142408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698399.142450:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142454:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698399.142465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698399.142472:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142481:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698399.142486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142510:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698399.142517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698399.142520:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698399.142524:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698399.142551:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698399.142571:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142576:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698399.142583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698399.142588:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698424 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142594:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698399.142597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698399/real 1787698399] req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142603:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698399.142609:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698399.142613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698399.142615:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698399.142617:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698399.142621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698399.142624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126333824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698399.142629:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698399.142633:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126333952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698399.142638:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698399.142641:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126334080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698399.142644:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698399.142647:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126334208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698424 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698399.142650:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698399.653556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698399.653556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698399.653556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698399.781635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698400.677547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698400.677547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698400.677769:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698400.805597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698401.061530:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300974447, at 4300974464 00000100:00000040:2.0:1787698401.061551:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300974447, at 4300974464 00000100:00100000:3.0:1787698401.701564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698401.701565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698401.701573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698401.829631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698402.725557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698402.725558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698402.725558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698402.853570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698403.749550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698403.749550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698403.749550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698403.877581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698404.261566:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698404.261575:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698404.261579:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698404.261589:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6305 00000100:00080000:1.0:1787698404.261601:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698404.261605:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261614:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698404.261620:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698404.261624:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261626:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261629:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698404.261632:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261647:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698404.261675:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698404.261687:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698404.261712:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698404.261716:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698404.261718:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698404.261722:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6305 00000100:00080000:1.0:1787698404.261726:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698404.261730:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261735:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261737:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698404.261739:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698404.261741:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261743:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261745:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698404.261748:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698404.261758:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698404.261760:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698404.261763:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698404.261780:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698404.261819:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698404.261831:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698404.261838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126334336:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698404.261839:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698404.261841:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698404.261852:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6305 00000100:00080000:1.0:1787698404.261856:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698404.261858:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261860:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261861:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698404.261863:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698404.261865:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261869:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261871:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698404.261874:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698404.261880:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698404.261884:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698404.261888:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698404.261901:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698404.261907:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698404.261918:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698404.261924:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698404.261927:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698404.261930:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698404.261934:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6305 00000100:00080000:1.0:1787698404.261940:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698404.261940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261951:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261957:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261960:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698404.261963:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698404.261968:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698404.261971:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.261979:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.261985:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698404.261994:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698404.262008:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698404.262014:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698404.262023:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698404.262031:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698404.262039:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698404.262040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698404/real 1787698404] req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698404.262044:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698404.262048:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698404.262057:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6305 00000100:00080000:1.0:1787698404.262067:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698404.262075:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.262079:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.262081:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698404.262083:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:2.0:1787698404.262088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698404.262090:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698404.262101:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698404.262107:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698404.262110:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698404.262113:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698404.262117:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698404.262127:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698404.262131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698404.262161:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698404.262162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698404.262178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698404.262180:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6315) 00000100:00000040:2.0:1787698404.262186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126334336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698404.262206:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126334336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698404.262217:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698404.262234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698404.262272:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262280:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698404.262292:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698404.262300:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262310:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698404.262316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698404/real 1787698404] req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698404.262343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698404.262347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698404.262350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698404.262363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698404.262384:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262391:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698404.262402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698404.262408:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262417:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698404.262445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698404/real 1787698404] req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262470:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262479:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698404.262488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698404.262491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698404.262495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698404.262509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698404.262530:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262535:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698404.262544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698404.262548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262554:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698404.262557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698404/real 1787698404] req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698404.262574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698404.262575:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698404.262577:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262581:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698404.262585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698404.262596:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262600:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698404.262605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698404.262609:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698429 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262614:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698404.262616:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698404/real 1787698404] req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698404.262627:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698404.262631:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698404.262632:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698404.262634:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698404.262638:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126334848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698404.262641:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126334464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698404.262645:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698404.262649:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126334592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698404.262654:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698404.262657:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126334720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698404.262660:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698404.262664:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126334848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698429 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698404.262668:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698404.773570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698404.773570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698404.773574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698404.901652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698405.797569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698405.797569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698405.797574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698405.925636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698406.181560:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300979567, at 4300979584 00000100:00000040:2.0:1787698406.181567:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300979567, at 4300979584 00000100:00100000:0.0:1787698406.821559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698406.821559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698406.821560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698406.949608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698407.845487:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698407.845530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698407.845531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698407.973615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698408.869537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698408.869537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698408.869544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698408.997596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698409.381534:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698409.381545:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698409.381549:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698409.381559:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6310 00000100:00080000:1.0:1787698409.381570:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698409.381572:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381576:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381579:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698409.381583:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698409.381586:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381588:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381590:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698409.381592:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381605:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698409.381640:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698409.381652:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698409.381671:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698409.381674:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698409.381675:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698409.381678:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6310 00000100:00080000:1.0:1787698409.381680:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698409.381682:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381684:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381685:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698409.381688:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698409.381691:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381693:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381694:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698409.381696:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698409.381704:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698409.381709:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698409.381717:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698409.381719:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698409.381793:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.381793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126334976:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698409.381813:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698409.381818:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698409.381821:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698409.381828:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6310 00000100:00080000:1.0:1787698409.381832:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698409.381836:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698409.381841:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698409.381843:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381854:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698409.381860:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698409.381870:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698409.381877:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381884:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381887:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698409.381891:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698409.381896:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698409.381900:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698409.381920:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698409.381920:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381932:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698409.381938:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698409.381942:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698409.381946:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698409.381947:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698409.381954:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6310 00000100:00080000:1.0:1787698409.381965:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698409.381971:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381975:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381978:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698409.381980:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698409.381985:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698409.381989:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.381991:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698409.381997:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698409.381998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698409/real 1787698409] req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698409.382007:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698409.382020:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698409.382034:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698409.382036:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698409.382075:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698409.382081:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698409.382115:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698409.382118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698409.382119:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698409.382124:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698409.382130:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787698409.382132:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6310 00000100:00080000:1.0:1787698409.382141:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698409.382146:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698409.382146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698409.382155:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.382158:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698409.382161:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698409.382164:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698409.382167:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698409.382177:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787698409.382180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126334976:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698409.382185:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698409.382198:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698409.382199:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126334976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698409.382207:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:2.0:1787698409.382212:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698409.382219:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698409.382229:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698409.382259:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6320) 00000100:00100000:2.0:1787698409.382260:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698409.382274:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382280:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698409.382290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698409.382295:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382302:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698409.382305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698409/real 1787698409] req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382317:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698409.382322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698409.382324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698409.382326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698409.382338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698409.382349:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382353:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698409.382358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698409.382361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382388:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698409.382392:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698409/real 1787698409] req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382399:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698409.382409:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698409.382410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698409.382412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382416:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698409.382438:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698409.382452:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382456:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698409.382462:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698409.382466:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382471:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698409.382475:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698409/real 1787698409] req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698409.382490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698409.382492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698409.382495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382502:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698409.382508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126335104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698409.382515:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698409.382521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126335232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698409.382527:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698409.382532:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126335360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698409.382539:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698409.382547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698409.382567:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382574:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698409.382586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698409.382592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698434 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382601:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698409.382606:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698409/real 1787698409] req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382617:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698409.382625:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698409.382633:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698409.382635:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698409.382637:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698409.382642:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698409.382645:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126335488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698434 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698409.382649:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698409.893521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698409.893522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698409.893588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698410.021624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698410.917599:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698410.917600:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698410.917601:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698411.045594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698411.301564:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300984687, at 4300984704 00000100:00000040:3.0:1787698411.301564:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300984687, at 4300984704 00000100:00100000:1.0:1787698411.941552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698411.941552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698411.941560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698412.069649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698412.965541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698412.965543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698412.965545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698413.093605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698413.989509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698413.989509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698413.989509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698414.117579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698414.501568:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698414.501579:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698414.501583:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698414.501591:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6315 00000100:00080000:2.0:1787698414.501600:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698414.501602:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501605:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501608:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698414.501612:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698414.501614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501615:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501618:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698414.501620:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501632:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698414.501651:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698414.501661:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698414.501680:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698414.501683:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698414.501684:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698414.501686:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6315 00000100:00080000:2.0:1787698414.501688:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698414.501690:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501691:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501692:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698414.501694:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698414.501696:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501698:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501700:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698414.501702:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698414.501708:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698414.501711:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698414.501718:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698414.501724:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698414.501727:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.501760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126335616:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698414.501771:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698414.501773:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698414.501775:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698414.501780:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6315 00000100:00080000:2.0:1787698414.501784:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698414.501786:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501789:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698414.501794:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698414.501794:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698414.501797:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698414.501805:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501813:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787698414.501816:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698414.501821:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698414.501826:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698414.501829:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698414.501837:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698414.501843:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698414.501849:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698414.501851:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698414.501854:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698414.501856:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698414.501859:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6315 00000100:00080000:2.0:1787698414.501863:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698414.501867:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501868:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501870:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698414.501872:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698414.501876:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698414.501878:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698414.501879:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501882:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698414.501886:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698414.501891:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698414.501897:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698414.501903:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698414.501909:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698414.501915:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698414.501917:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698414.501918:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698414.501921:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698414/real 1787698414] req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698414.501922:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6315 00000100:00080000:2.0:1787698414.501927:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698414.501930:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501935:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698414.501939:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698414.501942:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698414.501947:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698414.501951:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698414.501954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698414.501955:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698414.501962:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698414.501970:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698414.501975:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698414.501977:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698414.501989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698414.501990:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698414.501998:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698414.502004:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698414.502014:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6325) 00000100:00100000:1.0:1787698414.502022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126335616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698414.502031:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126335616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698414.502042:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698414.502057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502065:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698414.502077:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502083:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698414.502094:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698414.502101:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502110:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698414.502115:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698414/real 1787698414] req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502125:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502134:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698414.502141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698414.502144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698414.502147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698414.502159:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502166:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698414.502176:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502182:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698414.502190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698414.502196:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502208:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698414.502212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698414/real 1787698414] req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698414.502237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698414.502240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698414.502243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502249:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126335872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698414.502254:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502261:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698414.502272:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502299:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698414.502310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698414.502317:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502326:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698414.502333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698414/real 1787698414] req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698414.502361:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698414.502363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698414.502367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698414.502378:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502383:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698414.502392:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502397:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698414.502404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698414.502409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698439 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502414:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698414.502432:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698414/real 1787698414] req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502445:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698414.502450:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698414.502455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698414.502457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698414.502458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698414.502463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698414.502466:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126335744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698414.502471:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698414.502474:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126335872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698414.502479:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698414.502482:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126336000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698414.502486:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698414.502488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126336128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698439 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698414.502491:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698415.013502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698415.013502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698415.013523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698415.141577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698416.037603:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698416.037607:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698416.038171:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698416.165575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698416.421535:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300989807, at 4300989824 00000100:00000040:3.0:1787698416.421541:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300989807, at 4300989824 00000100:00100000:1.0:1787698417.061503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698417.061503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698417.061569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698417.189625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698418.085530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698418.085530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698418.085550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698418.213589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698419.109488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698419.109489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698419.109495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698419.237606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698419.621544:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698419.621556:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698419.621567:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698419.621578:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6320 00000100:00080000:2.0:1787698419.621589:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698419.621592:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621595:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621598:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698419.621603:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698419.621606:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621607:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621610:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698419.621612:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621630:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698419.621670:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698419.621683:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698419.621702:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698419.621705:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698419.621706:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698419.621709:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6320 00000100:00080000:2.0:1787698419.621713:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698419.621715:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621717:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621719:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698419.621721:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698419.621725:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621728:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698419.621731:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698419.621740:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698419.621749:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698419.621751:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698419.621759:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698419.621794:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.621796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126336256:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698419.621816:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698419.621817:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698419.621820:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698419.621825:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6320 00000100:00080000:2.0:1787698419.621827:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698419.621830:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621832:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621833:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787698419.621836:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698419.621836:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698419.621847:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621859:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698419.621860:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621866:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698419.621871:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698419.621878:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698419.621885:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698419.621896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698419.621901:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698419.621911:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.621917:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698419.621938:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698419.621945:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698419.621947:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698419.621952:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6320 00000100:00080000:2.0:1787698419.621955:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698419.621959:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621964:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621966:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:1.0:1787698419.621969:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698419/real 1787698419] req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698419.621969:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698419.621978:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698419.621980:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.621981:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698419.621983:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698419.621988:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698419.621993:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698419.621999:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698419.622004:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698419.622053:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698419.622062:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698419.622064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698419.622065:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698419.622069:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698419.622075:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6320 00000100:00080000:2.0:1787698419.622082:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698419.622090:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698419.622092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698419.622100:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698419.622100:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698419.622111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698419.622111:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698419.622120:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698419.622126:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126336256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698419.622132:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698419.622139:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126336256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698419.622146:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698419.622157:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000020:00000040:1.0:1787698419.622161:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:2.0:1787698419.622166:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698419.622173:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698419.622179:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698419.622180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698419.622215:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698419.622222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698419.622251:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622254:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622320:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698419.622325:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6330) 00000800:00000100:1.0:1787698419.622336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698419.622345:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622352:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698419.622356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698419/real 1787698419] req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698419.622374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698419.622376:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698419.622379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698419.622387:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698419.622399:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622403:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698419.622408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698419.622412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622441:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698419.622445:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698419/real 1787698419] req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622453:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698419.622463:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698419.622465:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698419.622467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622475:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698419.622479:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622484:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698419.622492:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622496:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698419.622501:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698419.622505:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622511:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698419.622513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698419/real 1787698419] req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622520:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622524:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698419.622528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698419.622530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698419.622532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698419.622539:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126336384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698419.622544:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698419.622548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126336512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698419.622552:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698419.622556:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126336640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698419.622560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698419.622564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698419.622575:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622579:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698419.622584:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698419.622587:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698444 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622592:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698419.622595:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698419/real 1787698419] req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698419.622606:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698419.622610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698419.622612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698419.622614:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698419.622618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126336768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698419.622621:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126336768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698444 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698419.622624:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698420.133520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698420.133559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698420.133561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698420.261550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698421.157529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698421.157529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698421.157529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698421.285628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698421.541539:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4300994927, at 4300994944 00000100:00000040:3.0:1787698421.541539:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4300994927, at 4300994944 00000100:00100000:1.0:1787698422.181553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698422.181553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698422.181580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698422.309617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698423.205554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698423.205554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698423.205557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698423.333615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698424.229549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698424.229583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698424.229626:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698424.357630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698424.741571:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698424.741586:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698424.741592:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698424.741607:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6325 00000100:00080000:2.0:1787698424.741621:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698424.741626:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.741631:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.741635:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698424.741642:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698424.741646:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.741649:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.741653:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698424.741657:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698424.741677:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698424.741718:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698424.741736:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698424.741765:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698424.741769:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698424.741772:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698424.741776:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6325 00000100:00080000:2.0:1787698424.741780:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698424.741783:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.741787:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.741790:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698424.741793:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698424.741797:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.741801:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.741804:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698424.741807:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698424.741819:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698424.741821:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698424.741827:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698424.741842:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698424.741846:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126336896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698424.741864:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698424.741876:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698424.741880:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698424.741882:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698424.741889:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6325 00000100:00000040:1.0:1787698424.741890:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698424.741895:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698424.741903:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.741911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698424.741913:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698424.741917:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698424.741921:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698424.741926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.741929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.741932:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698424.741936:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698424.741942:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698424.741949:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698424.741968:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698424.741974:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698424.741989:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.741995:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698424.742008:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698424.742014:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698424.742016:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698424.742023:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6325 00000100:00080000:2.0:1787698424.742027:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698424.742031:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.742034:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.742037:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698424.742040:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698424.742045:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698424/real 1787698424] req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698424.742048:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.742056:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.742062:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698424.742068:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698424.742075:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698424.742080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698424.742084:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698424.742114:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698424.742117:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698424.742141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698424.742146:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698424.742151:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698424.742162:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698424.742165:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698424.742175:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698424.742180:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698424.742186:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6325 00000100:00080000:2.0:1787698424.742190:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698424.742195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126336896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698424.742198:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.742206:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.742210:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787698424.742213:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126336896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698424.742214:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698424.742223:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698424.742230:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698424.742236:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000020:00000040:1.0:1787698424.742239:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:2.0:1787698424.742240:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698424.742250:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698424.742256:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698424.742263:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742275:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337024:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787698424.742285:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698424.742297:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698424.742303:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742318:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698424.742329:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6335) 00000800:00000100:1.0:1787698424.742340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698424.742354:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742372:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698424.742379:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698424/real 1787698424] req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698424.742410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698424.742413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698424.742455:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698424.742472:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698424.742495:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742503:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698424.742516:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698424.742524:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742535:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698424.742543:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698424/real 1787698424] req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742572:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698424.742580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698424.742584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698424.742588:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742596:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698424.742602:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742611:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698424.742624:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742632:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698424.742644:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698424.742652:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742661:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698424.742667:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698424/real 1787698424] req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742695:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698424.742703:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698424.742706:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698424.742709:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698424.742724:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126337024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698424.742745:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698424.742753:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126337152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698424.742761:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698424.742767:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126337280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698424.742775:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698424.742784:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698424.742809:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742817:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698424.742831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698424.742839:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698449 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742849:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698424.742855:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698424/real 1787698424] req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742867:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698424.742876:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698424.742883:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698424.742887:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698424.742891:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698424.742899:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698424.742906:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126337408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698449 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698424.742915:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698425.253546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698425.253546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698425.253546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698425.381612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698426.277559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698426.277567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698426.277591:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698426.405541:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698426.661524:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301000047, at 4301000064 00000100:00000040:2.0:1787698426.661550:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301000047, at 4301000064 00000100:00100000:0.0:1787698427.301541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698427.301544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698427.301582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698427.429541:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698428.325565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698428.325565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698428.325598:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698428.453564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698429.349556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698429.349556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698429.349574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698429.477568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698429.861555:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698429.861569:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698429.861575:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698429.861589:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6330 00000100:00080000:2.0:1787698429.861611:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698429.861616:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.861621:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698429.861625:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698429.861632:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698429.861636:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.861639:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698429.861643:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698429.861648:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698429.861666:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698429.861717:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698429.861734:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698429.861754:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698429.861758:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698429.861760:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698429.861763:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6330 00000100:00080000:2.0:1787698429.861767:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698429.861769:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.861771:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698429.861773:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698429.861776:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698429.861779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.861784:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698429.861786:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698429.861788:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698429.861798:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698429.861807:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698429.861811:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698429.861830:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698429.861833:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126337536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698429.861858:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698429.861866:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698429.861869:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698429.861875:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698429.861879:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6330 00000100:00080000:2.0:1787698429.861887:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698429.861888:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698429.861900:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.861912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698429.861919:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698429.861921:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698429.861925:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698429.861928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.861930:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698429.861933:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698429.861935:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698429.861940:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698429.861947:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698429.861955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698429.861957:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698429.861973:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698429.861975:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862038:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698429/real 1787698429] req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698429.862050:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698429.862056:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698429.862059:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698429.862060:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698429.862076:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6330 00000100:00000040:1.0:1787698429.862079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698429.862084:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698429.862088:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698429.862093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698429.862094:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698429.862102:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698429.862107:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698429.862115:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698429.862120:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698429.862133:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698429.862134:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126337536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698429.862139:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698429.862142:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698429.862151:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698429.862161:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126337536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698429.862161:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698429.862178:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787698429.862196:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698429.862203:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698429.862216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698429.862221:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337664:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698429.862261:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698429.862268:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698429.862272:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698429.862273:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698429.862288:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6330 00000100:00000040:1.0:1787698429.862294:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698429.862301:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698429.862310:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.862318:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698429.862320:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698429.862322:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698429.862327:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698429.862337:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698429.862341:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698429.862355:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698429.862359:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:2.0:1787698429.862361:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698429.862367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698429/real 1787698429] req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698429.862372:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698429.862391:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698429.862396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698429.862403:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698429.862408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698429.862414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698429.862450:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698429.862457:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698429.862474:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862475:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698429.862542:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6340) 00000100:00000040:1.0:1787698429.862553:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862561:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698429.862573:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862578:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698429.862587:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698429.862592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862597:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698429.862601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698429/real 1787698429] req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862612:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698429.862617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698429.862619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698429.862621:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862625:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698429.862628:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126337664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698429.862634:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698429.862638:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126337792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698429.862642:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698429.862648:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698429.862660:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862665:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698429.862670:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698429.862674:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862679:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698429.862682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698429/real 1787698429] req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862688:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862693:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698429.862697:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698429.862698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698429.862700:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862711:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126337920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698429.862715:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698429.862726:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862730:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698429.862736:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698429.862739:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698454 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862744:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698429.862747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698429/real 1787698429] req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698429.862758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698429.862761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698429.862763:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698429.862765:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698429.862769:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698429.862772:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126337920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698429.862777:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698429.862780:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126338048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698454 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698429.862783:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698430.373527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698430.373542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698430.373548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698430.501760:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698431.397568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698431.397569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698431.397573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698431.525588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698431.781543:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301005167, at 4301005184 00000100:00000040:3.0:1787698431.781543:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301005167, at 4301005184 00000100:00100000:2.0:1787698432.421535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698432.421535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698432.421580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698432.549606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698433.445566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698433.445566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698433.445566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698433.573586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698434.469496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698434.469497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698434.469503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698434.597707:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698434.981671:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698434.981681:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698434.981685:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698434.981697:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6335 00000100:00080000:0.0:1787698434.981711:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698434.981715:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698434.981721:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.981725:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698434.981730:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698434.981734:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698434.981737:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.981741:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698434.981745:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698434.981763:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698434.981800:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698434.981820:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698434.981868:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698434.981872:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698434.981874:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698434.981880:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6335 00000100:00080000:0.0:1787698434.981886:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698434.981889:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698434.981891:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.981894:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698434.981896:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698434.981899:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698434.981905:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698434.981906:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.981910:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698434.981916:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698434.981941:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698434.981943:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126338176:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698434.981947:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698434.981960:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698434.981969:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698434.981977:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698434.981980:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:1.0:1787698434.981980:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698434.981984:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698434.981993:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6335 00000100:00080000:0.0:1787698434.981998:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698434.981999:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698434.982015:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698434.982017:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.982019:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698434.982021:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698434.982024:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698434.982027:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.982029:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698434.982031:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698434.982035:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698434.982039:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698434.982041:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698434.982052:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698434.982060:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698434.982062:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698434.982087:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698434.982091:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698434.982093:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698434.982100:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6335 00000100:00080000:0.0:1787698434.982106:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698434.982107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698434/real 1787698434] req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698434.982114:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698434.982122:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.982126:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698434.982129:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698434.982131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698434.982135:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698434.982144:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.982151:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698434.982158:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698434.982168:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698434.982172:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:1.0:1787698434.982173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698434.982180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698434.982184:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698434.982188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126338176:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698434.982193:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698434.982197:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126338176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698434.982207:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698434.982210:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698434.982224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698434.982226:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338304:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698434.982249:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698434.982255:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:1.0:1787698434.982258:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698434.982259:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698434.982270:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6335 00000100:00000040:1.0:1787698434.982272:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698434.982280:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698434.982287:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698434.982288:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698434.982296:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698434.982297:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698434.982302:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698434.982308:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698434.982311:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:0.0:1787698434.982316:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698434.982320:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698434/real 1787698434] req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698434.982324:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698434.982331:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698434.982338:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698434.982346:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698434.982354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698434.982355:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698434.982370:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698434.982375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698434.982393:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698434.982393:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698434.982397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698434.982404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982439:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698434.982444:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6346) 00000100:00000040:1.0:1787698434.982448:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698434.982469:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982474:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698434.982481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698434.982486:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982493:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698434.982497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698434/real 1787698434] req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982516:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698434.982521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698434.982523:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698434.982526:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982531:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698434.982534:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126338304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698434.982540:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698434.982552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126338432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698434.982560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698434.982566:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982572:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698434.982581:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982586:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698434.982593:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698434.982597:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982604:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698434.982608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698434/real 1787698434] req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982621:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698434.982626:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698434.982628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698434.982631:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698434.982638:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698434.982650:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982655:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698434.982660:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698434.982664:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698459 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982674:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698434.982678:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698434/real 1787698434] req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982685:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698434.982691:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698434.982695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698434.982697:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698434.982699:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698434.982704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698434.982707:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126338560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698434.982712:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698434.982716:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126338688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698459 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698434.982721:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698435.493551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698435.493551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698435.493555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698435.621578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698436.517488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698436.517508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698436.517511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698436.645606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698436.901526:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301010287, at 4301010304 00000100:00000040:3.0:1787698436.902164:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301010287, at 4301010304 00000100:00100000:0.0:1787698437.541550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698437.541550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698437.541698:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698437.669546:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698438.565541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698438.565541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698438.565575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698438.693571:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698439.589582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698439.589582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698439.589590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698439.717561:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698440.101527:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.101537:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698440.101542:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.101552:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6341 00000100:00080000:3.0:1787698440.101562:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698440.101565:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101569:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101572:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698440.101576:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698440.101580:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101582:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101585:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698440.101588:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101601:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698440.101625:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698440.101641:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698440.101674:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.101678:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698440.101680:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.101684:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6341 00000100:00080000:3.0:1787698440.101688:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698440.101698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101705:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101708:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698440.101711:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698440.101716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.101717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101723:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101728:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698440.101733:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698440.101745:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698440.101747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126338816:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698440.101752:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698440.101766:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698440.101775:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.101780:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.101786:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698440.101795:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.101799:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698440.101801:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698440.101806:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6341 00000100:00080000:3.0:1787698440.101811:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698440.101819:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101820:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101828:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698440.101831:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698440.101835:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.101837:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101856:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101861:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698440.101863:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698440.101869:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698440.101874:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698440.101884:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698440.101887:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698440.101903:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.101906:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.101917:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698440.101925:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698440.101927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698440.101931:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698440.101933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698440.101935:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698440.101938:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698440.101943:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6341 00000100:00100000:1.0:1787698440.101950:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126338816:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698440.101951:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698440.101958:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698440.101959:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126338816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.101965:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101969:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698440.101973:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000020:00000040:1.0:1787698440.101974:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101977:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.101982:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.101984:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698440.101994:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698440.101994:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698440.102003:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698440.102009:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698440.102021:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338944:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787698440.102023:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698440.102037:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102040:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102060:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698440.102060:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698440.102065:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698440.102067:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698440.102071:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698440.102074:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6341 00000100:00080000:3.0:1787698440.102079:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698440.102080:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.102086:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.102092:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698440.102094:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:3.0:1787698440.102096:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698440.102100:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698440.102102:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.102105:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698440.102112:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698440.102115:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698440.102118:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698440.102122:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698440.102130:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698440.102133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698440.102137:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698440.102139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698440.102143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698440.102146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698440.102150:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698440.102160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126338944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698440.102169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698440.102174:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698440.102195:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6351) 00000100:00000040:1.0:1787698440.102200:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102209:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698440.102221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698440.102228:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102237:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698440.102242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698440.102263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698440.102266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698440.102269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698440.102281:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126338944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698440.102289:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698440.102295:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126339072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698440.102308:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698440.102314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698440.102328:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102333:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698440.102340:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698440.102344:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102350:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698440.102353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698440.102369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698440.102370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698440.102372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698440.102383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698440.102400:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102406:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698440.102416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698440.102452:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698465 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102460:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698440.102465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698440/real 1787698440] req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698440.102481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698440.102487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698440.102490:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698440.102494:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698440.102501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698440.102512:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126339200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698440.102525:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698440.102531:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126339328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698465 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698440.102537:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698440.613580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698440.613580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698440.613600:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698440.741557:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698441.637522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698441.637527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698441.637560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698441.765588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698442.021522:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301015407, at 4301015424 00000100:00000040:3.0:1787698442.021525:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301015407, at 4301015424 00000100:00100000:1.0:1787698442.661534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698442.661534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698442.661539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698442.789561:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698443.685766:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698443.685813:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698443.686213:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698443.813606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698444.709656:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698444.709977:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698444.710922:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698444.837670:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698445.221527:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698445.221537:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698445.221541:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698445.221552:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6346 00000100:00080000:3.0:1787698445.221564:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698445.221567:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.221571:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221575:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698445.221580:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698445.221582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.221584:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221587:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698445.221590:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221604:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698445.221631:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698445.221644:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698445.221669:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698445.221672:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698445.221674:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698445.221692:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6346 00000100:00080000:3.0:1787698445.221695:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698445.221698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.221700:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221703:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698445.221705:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698445.221708:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.221710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221712:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698445.221714:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698445.221723:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698445.221724:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698445.221731:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698445.221745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126339456:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698445.221745:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698445.221781:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698445.221789:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698445.221792:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698445.221793:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698445.221796:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6346 00000100:00080000:3.0:1787698445.221800:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698445.221801:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698445.221806:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.221813:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221818:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787698445.221822:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221823:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698445.221828:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.221841:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221844:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698445.221846:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698445.221852:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698445.221858:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698445.221865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698445.221872:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698445.221885:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698445.221888:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698445.221908:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698445.221911:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698445.221913:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698445.221916:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6346 00000100:00080000:3.0:1787698445.221920:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698445.221923:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.221925:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221927:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698445.221929:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698445.221935:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698445.221937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698445/real 1787698445] req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698445.221938:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.221941:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698445.221946:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698445.221953:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698445.221958:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698445.221966:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698445.221971:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698445.221993:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.221998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698445.222010:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698445.222018:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698445.222021:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698445.222029:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6346 00000100:00080000:3.0:1787698445.222035:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698445.222036:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698445.222037:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698445.222043:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.222047:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:1.0:1787698445.222048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:3.0:1787698445.222049:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698445.222056:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698445.222060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698445.222061:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698445.222065:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698445.222070:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698445.222078:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698445.222083:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698445.222090:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126339456:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698445.222090:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698445.222111:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222115:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126339456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698445.222119:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6356) 00000020:00000040:1.0:1787698445.222134:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698445.222154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222162:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698445.222176:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222184:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698445.222198:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698445.222206:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222219:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698445.222226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698445/real 1787698445] req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222235:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698445.222252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698445.222254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698445.222257:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222261:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698445.222265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698445.222281:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222287:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698445.222295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698445.222299:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222306:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698445.222309:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698445/real 1787698445] req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222317:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698445.222329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698445.222331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698445.222334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698445.222341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698445.222354:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222358:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698445.222364:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698445.222368:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222374:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698445.222378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698445/real 1787698445] req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698445.222395:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698445.222397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698445.222400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698445.222408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222413:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698445.222445:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222454:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698445.222465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698445.222473:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698470 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222493:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698445.222500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698445/real 1787698445] req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698445.222517:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698445.222522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698445.222524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698445.222526:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698445.222532:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126339968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698445.222536:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126339584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698445.222541:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698445.222545:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126339712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698445.222549:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698445.222553:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126339840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698445.222558:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698445.222569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126339968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698470 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698445.222574:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698445.733582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698445.733583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698445.733594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698445.861587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698446.757519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698446.757523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698446.757525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698446.885644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698447.141588:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301020527, at 4301020544 00000100:00000040:3.0:1787698447.141588:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301020527, at 4301020544 00000100:00100000:0.0:1787698447.781550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698447.781551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698447.781666:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698447.909557:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698448.805509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698448.805509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698448.805568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698448.933604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698449.829571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698449.829571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698449.829582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698449.957565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698450.341538:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698450.341549:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698450.341554:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698450.341568:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6351 00000100:00080000:3.0:1787698450.341584:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698450.341587:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341592:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341596:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698450.341602:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698450.341606:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341608:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341611:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698450.341615:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341634:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698450.341685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698450.341703:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698450.341730:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698450.341734:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698450.341735:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698450.341740:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6351 00000100:00080000:3.0:1787698450.341744:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698450.341768:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341771:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341774:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:1.0:1787698450.341780:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698450.341780:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698450.341790:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341800:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00100000:1.0:1787698450.341802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126340096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698450.341807:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698450.341821:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698450.341828:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698450.341834:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698450.341844:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698450.341845:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341857:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698450.341866:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698450.341869:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698450.341870:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698450.341873:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6351 00000100:00080000:3.0:1787698450.341878:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698450.341880:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341884:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698450.341886:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:1.0:1787698450.341889:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341893:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341897:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341899:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787698450.341911:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698450.341912:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698450.341920:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698450.341925:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698450.341935:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698450.341943:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.341949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698450/real 1787698450] req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698450.341952:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698450.341960:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698450.341964:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698450.341970:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6351 00000100:00080000:3.0:1787698450.341973:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698450.341977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698450.341979:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.341985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.341989:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698450.341993:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698450.341999:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698450.342000:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698450.342003:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.342007:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698450.342012:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698450.342020:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698450.342021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698450.342028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:3.0:1787698450.342029:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698450.342033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126340096:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698450.342044:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698450.342049:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126340096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698450.342072:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698450.342074:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342090:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698450.342095:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698450.342099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340224:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698450.342113:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698450.342116:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698450.342121:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698450.342126:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6351 00000100:00000040:1.0:1787698450.342136:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698450.342136:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698450.342143:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698450.342146:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698450.342151:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698450.342151:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698450.342155:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698450.342160:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698450.342162:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698450.342166:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698450.342170:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698450.342176:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698450.342183:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698450.342184:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:3.0:1787698450.342190:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698450.342193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698450/real 1787698450] req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698450.342202:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698450.342205:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698450.342230:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698450.342237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698450.342240:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698450.342246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342266:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698450.342266:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6361) 00000100:00000040:1.0:1787698450.342277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698450.342304:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342313:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698450.342327:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698450.342335:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342349:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698450.342353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698450/real 1787698450] req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342361:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342368:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698450.342373:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698450.342375:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698450.342377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698450.342387:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126340224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698450.342392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698450.342396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126340352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698450.342400:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698450.342406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342412:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698450.342450:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342456:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698450.342464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698450.342469:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342475:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698450.342479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698450/real 1787698450] req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342493:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698450.342498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698450.342500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698450.342502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698450.342510:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698450.342523:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342533:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698450.342538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698450.342542:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698475 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342548:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698450.342552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698450/real 1787698450] req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698450.342565:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698450.342569:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698450.342571:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698450.342573:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698450.342578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698450.342582:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126340480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698450.342587:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698450.342590:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126340608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698475 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698450.342594:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698450.853535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698450.853536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698450.853548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698450.981600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698451.877538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698451.877538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698451.877551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698452.005717:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698452.261542:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301025647, at 4301025664 00000100:00000040:2.0:1787698452.261561:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301025647, at 4301025664 00000100:00100000:1.0:1787698452.901572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698452.901574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698452.901804:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698453.029653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698453.925547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698453.925598:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698453.925627:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698454.053648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698454.949552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698454.949602:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698454.949606:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698455.077615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698455.461536:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698455.461551:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698455.461597:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698455.461612:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6356 00000100:00080000:3.0:1787698455.461625:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698455.461630:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.461635:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698455.461640:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698455.461645:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698455.461650:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.461653:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698455.461657:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698455.461660:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698455.461678:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698455.461711:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698455.461729:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698455.461756:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698455.461761:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698455.461763:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698455.461768:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6356 00000100:00080000:3.0:1787698455.461773:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698455.461777:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.461781:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698455.461784:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698455.461788:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698455.461792:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.461796:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698455.461801:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698455.461813:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698455.461820:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698455.461822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126340736:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698455.461833:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698455.461850:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698455.461856:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698455.461867:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698455.461868:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698455.461883:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698455.461896:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698455.461908:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698455.461910:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698455.461915:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6356 00000100:00080000:3.0:1787698455.461921:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698455.461926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.461930:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698455.461933:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698455.461937:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698455.461944:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.461950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698455.461952:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698455.461955:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698455.461965:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698455.461968:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340864:192.168.201.136@tcp:38:kworker.0 00000008:00100000:3.0:1787698455.461973:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698455.461982:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698455.461984:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698455.461994:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698455.462004:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698455.462012:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787698455.462034:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698455.462054:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698455.462059:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698455.462061:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698455.462068:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6356 00000100:00080000:3.0:1787698455.462074:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698455.462078:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.462092:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698455.462095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698455.462096:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698455.462103:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698455.462110:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698455.462111:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.462119:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698455.462123:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698455.462127:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698455.462129:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698455.462139:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698455.462139:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698455.462149:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698455.462159:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787698455.462165:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698455.462178:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698455.462194:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698455.462198:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698455.462201:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698455.462211:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6356 00000100:00080000:3.0:1787698455.462217:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698455.462220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698455.462224:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.462232:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698455.462237:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698455.462241:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698455.462245:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698455.462248:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698455.462250:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698455.462254:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698455.462256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341120:192.168.201.136@tcp:8:kworker.0 00000008:00100000:3.0:1787698455.462260:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698455.462267:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698455.462279:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698455.462281:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698455.462309:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698455.462314:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698455.462326:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787698455.462330:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6366) 00000100:00000040:1.0:1787698455.462344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698455.462356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698455.462372:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698455.462382:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698455.462395:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698455.973559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698455.973581:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698455.973590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698456.101683:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698456.997530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698456.997566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698456.997590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698457.125607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698457.381495:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301030767, at 4301030784 00000100:00000040:2.0:1787698457.381504:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301030767, at 4301030784 00000100:00100000:1.0:1787698458.021557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698458.021569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698458.021619:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698458.149623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698458.533822:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698458.533878:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698458.533896:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698458.533946:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698458.533981:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698458.533992:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698458.534023:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698458.534034:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698458.534045:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698458.534055:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698458.534066:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698458.534076:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698480 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698455/real 1787698458] req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698458.534206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698458.534218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698458.534237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698458.534244:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341248:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698458.534261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698455/real 1787698458] req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534274:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698458.534284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698458.534292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698458.534295:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698458.534300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341120:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698458.534315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698455/real 1787698458] req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698458.534336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698458.534350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698458.534355:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698458.534364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340992:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698458.534378:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698455/real 1787698458] req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698458.534399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698458.534406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698458.534409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698458.534414:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126340864:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698458.534463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698455/real 1787698458] req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698458.534486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698458.534500:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698458.534503:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698458.534508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698458.534515:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126340736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698458.534521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126341248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698458.534549:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698458.534562:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126341120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698458.534570:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698458.534574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126340992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698458.534587:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698458.534592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126340864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698458.534600:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698458.534605:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126340736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698480 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698458.534612:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698459.045570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698459.045570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698459.045570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698459.173591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698460.069532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698460.069538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698460.069788:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698460.197651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698460.581564:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698460.581576:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698460.581581:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698460.581593:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6361 00000100:00080000:1.0:1787698460.581604:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698460.581607:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581611:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581615:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698460.581621:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698460.581624:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581626:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581629:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698460.581632:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581644:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698460.581671:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698460.581683:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698460.581708:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698460.581710:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698460.581711:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698460.581714:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6361 00000100:00080000:1.0:1787698460.581717:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698460.581719:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581721:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581723:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698460.581725:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698460.581728:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581731:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581733:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698460.581736:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698460.581747:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698460.581764:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698460.581772:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698460.581783:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698460.581796:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698460.581810:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698460.581815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126341376:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698460.581816:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698460.581820:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698460.581829:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6361 00000100:00080000:1.0:1787698460.581834:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698460.581837:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581839:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581842:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698460.581844:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698460.581848:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581857:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581860:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787698460.581864:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698460.581866:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698460.581876:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698460.581883:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581885:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698460.581899:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698460.581908:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698460.581914:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698460.581918:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698460.581920:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698460.581931:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6361 00000100:00080000:1.0:1787698460.581936:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698460.581940:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581943:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581949:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.581952:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698460.581955:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698460.581959:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.581966:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698460.581967:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698460.581971:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698460.581977:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698460.581988:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698460.581995:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698460.582008:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698460.582019:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698460/real 1787698460] req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698460.582034:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698460.582043:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698460.582048:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698460.582055:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6361 00000100:00080000:1.0:1787698460.582060:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698460.582063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698460.582066:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698460.582074:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.582080:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698460.582085:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698460.582093:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698460.582094:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698460.582098:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698460.582104:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698460.582111:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787698460.582115:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787698460.582119:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698460.582125:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787698460.582128:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698460.582135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126341376:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698460.582150:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698460.582165:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126341376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698460.582171:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698460.582189:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698460.582191:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6371) 00000100:00000040:2.0:1787698460.582218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698460.582255:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582263:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698460.582279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698460.582288:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582299:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698460.582306:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698460/real 1787698460] req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698460.582336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698460.582339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698460.582343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698460.582358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582367:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698460.582382:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582391:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698460.582403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698460.582409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582416:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698460.582451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698460/real 1787698460] req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698460.582474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698460.582477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698460.582481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698460.582516:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698460.582544:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582551:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698460.582562:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698460.582569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582580:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698460.582586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698460/real 1787698460] req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582598:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582606:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698460.582613:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698460.582617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698460.582621:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698460.582635:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582644:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698460.582657:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582664:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698460.582676:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698460.582684:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698485 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582693:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698460.582699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698460/real 1787698460] req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698460.582718:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698460.582726:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698460.582730:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698460.582734:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698460.582743:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126341888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698460.582749:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126341504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698460.582759:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698460.582767:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126341632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698460.582775:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698460.582782:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126341760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698460.582791:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698460.582798:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126341888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698485 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698460.582807:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698461.093536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698461.093548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698461.093548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698461.221620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698462.117587:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698462.117598:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698462.117599:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698462.245565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698462.501535:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301035887, at 4301035904 00000100:00000040:2.0:1787698462.501544:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301035887, at 4301035904 00000100:00100000:3.0:1787698463.141536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698463.141543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698463.141595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698463.269599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698464.165517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698464.165544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698464.165564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698464.293728:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698465.189573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698465.189579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698465.189580:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698465.317637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698465.701579:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698465.701589:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698465.701594:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698465.701605:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6366 00000100:00080000:0.0:1787698465.701617:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698465.701619:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698465.701623:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.701626:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698465.701631:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698465.701633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698465.701635:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.701638:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698465.701641:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698465.701656:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698465.701706:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698465.701750:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698465.701775:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698465.701780:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698465.701782:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698465.701787:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6366 00000100:00080000:0.0:1787698465.701793:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698465.701797:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698465.701810:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.701826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126342016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698465.701840:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.701844:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698465.701848:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698465.701869:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698465.701870:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698465.701875:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.701880:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698465.701887:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698465.701892:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787698465.701902:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698465.701910:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698465.701928:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698465.701934:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698465.701946:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.701955:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698465.701956:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698465.701966:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698465.701969:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698465.701983:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6366 00000100:00100000:2.0:1787698465.702002:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698465/real 1787698465] req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702022:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702035:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698465.702039:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698465.702047:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698465.702051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698465.702052:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.702058:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00080000:2.0:1787698465.702062:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787698465.702063:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698465.702071:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698465.702073:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698465.702077:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.702082:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698465.702089:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698465.702096:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126342016:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787698465.702098:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698465.702108:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698465.702111:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126342016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698465.702123:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787698465.702126:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698465.702139:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698465.702152:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698465.702152:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698465.702158:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698465.702162:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698465.702171:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6366 00000100:00100000:2.0:1787698465.702176:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342144:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698465.702177:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698465.702182:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698465.702186:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.702188:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698465.702190:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698465.702192:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698465.702202:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698465.702204:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.702209:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698465.702214:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698465.702222:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698465.702227:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698465.702245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698465.702254:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698465.702260:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698465.702268:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698465.702277:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698465.702280:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698465/real 1787698465] req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702305:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698465.702330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698465.702335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698465.702339:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702348:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698465.702355:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126342144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698465.702358:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698465.702376:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00080000:0.0:1787698465.702377:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698465.702383:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698465.702384:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698465.702389:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6366 00000100:00080000:0.0:1787698465.702392:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698465.702394:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698465.702397:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698465.702414:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.702442:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698465.702445:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698465.702449:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698465.702452:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698465.702454:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698465.702458:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698465.702465:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698465.702473:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698465.702475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698465.702494:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702503:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698465.702518:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698465.702520:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698465.702530:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698465.702540:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702551:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:0.0:1787698465.702554:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6376) 00000100:00100000:2.0:1787698465.702563:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698465/real 1787698465] req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702589:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698465.702605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698465.702609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698465.702614:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698465.702630:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702640:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698465.702656:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702665:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698465.702678:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698465.702686:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702697:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698465.702704:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698465/real 1787698465] req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702717:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698465.702736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698465.702740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698465.702745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698465.702759:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126342272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698465.702769:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698465.702776:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126342400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698465.702784:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698465.702793:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698465.702817:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702825:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698465.702838:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698465.702846:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698490 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702857:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698465.702864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698465/real 1787698465] req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698465.702888:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698465.702896:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698465.702900:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698465.702905:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698465.702914:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698465.702921:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126342528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698490 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698465.702930:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698466.213486:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698466.213502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698466.213504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698466.341676:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698467.237551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698467.237552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698467.237847:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698467.365583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698467.621520:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301041007, at 4301041024 00000100:00000040:3.0:1787698467.621520:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301041007, at 4301041024 00000100:00100000:3.0:1787698468.261535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698468.261554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698468.261587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698468.389560:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698469.285570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698469.285570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698469.285665:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698469.413630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698470.309552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698470.309552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698470.309552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698470.437551:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698470.821528:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698470.821538:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698470.821543:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698470.821552:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6371 00000100:00080000:1.0:1787698470.821563:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698470.821566:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821570:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821574:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698470.821578:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698470.821582:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821585:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821588:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698470.821591:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821605:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698470.821629:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698470.821641:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698470.821663:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698470.821666:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698470.821668:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698470.821673:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6371 00000100:00080000:1.0:1787698470.821677:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698470.821680:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821683:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821686:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698470.821688:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698470.821691:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821693:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821695:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698470.821697:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698470.821705:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698470.821709:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698470.821717:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698470.821725:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698470.821732:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698470.821734:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698470.821737:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698470.821745:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6371 00000100:00000040:2.0:1787698470.821745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698470.821755:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698470.821767:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821777:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821782:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698470.821784:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698470.821788:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821790:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821791:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698470.821793:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698470.821798:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698470.821800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126342656:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698470.821803:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698470.821820:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698470.821827:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698470.821833:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698470.821834:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698470.821836:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698470.821839:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6371 00000100:00080000:1.0:1787698470.821842:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698470.821844:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821847:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821848:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698470.821849:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698470.821852:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821859:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698470.821859:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698470.821864:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698470.821872:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698470.821877:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698470.821885:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698470.821899:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698470.821902:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821913:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698470.821922:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698470.821925:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698470.821927:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698470.821929:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6371 00000100:00080000:1.0:1787698470.821931:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698470.821933:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821934:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821935:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698470.821937:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698470.821939:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821940:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698470.821942:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698470.821944:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698470.821947:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698470.821950:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698470.821954:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698470.821966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698470.821968:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698470.821978:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6381) 00000100:00000040:2.0:1787698470.822004:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698470/real 1787698470] req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822056:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822063:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698470.822071:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698470.822074:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698470.822077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126342656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698470.822089:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126342656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698470.822097:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698470.822113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698470.822137:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822145:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698470.822159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698470.822166:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822179:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698470.822184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698470/real 1787698470] req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822198:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698470.822204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698470.822206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698470.822208:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822213:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698470.822218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822224:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698470.822232:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822237:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698470.822245:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698470.822250:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822256:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698470.822261:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698470/real 1787698470] req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698470.822279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698470.822281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698470.822284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126342912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698470.822297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698470.822336:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822345:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698470.822354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698470.822360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822367:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698470.822371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698470/real 1787698470] req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698470.822396:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698470.822398:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698470.822400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698470.822409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698470.822457:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822463:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698470.822470:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698470.822475:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698495 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822481:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698470.822485:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698470/real 1787698470] req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698470.822498:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698470.822503:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698470.822505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698470.822507:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698470.822512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698470.822516:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126342784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698470.822521:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698470.822525:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126342912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698470.822530:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698470.822533:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126343040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698470.822538:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698470.822541:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126343168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698495 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698470.822546:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698471.333531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698471.333531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698471.333531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698471.461594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698472.357614:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698472.357776:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698472.357905:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698472.485587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698472.741532:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301046127, at 4301046144 00000100:00000040:3.0:1787698472.741532:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301046127, at 4301046144 00000100:00100000:3.0:1787698473.381523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698473.381899:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698473.382247:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698473.509572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698474.405538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698474.405538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698474.405538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698474.533583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698475.429546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698475.429546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698475.429546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698475.557725:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698475.941545:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698475.941557:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698475.941561:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698475.941571:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6376 00000100:00080000:1.0:1787698475.941584:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698475.941586:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941590:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941592:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698475.941597:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698475.941599:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941601:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941603:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698475.941606:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941621:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698475.941650:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698475.941663:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698475.941680:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698475.941682:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698475.941683:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698475.941687:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6376 00000100:00080000:1.0:1787698475.941689:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698475.941692:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941694:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941696:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698475.941698:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698475.941700:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941702:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941704:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698475.941705:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698475.941713:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698475.941714:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698475.941717:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698475.941727:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698475.941737:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.941737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126343296:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698475.941749:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698475.941751:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698475.941752:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698475.941754:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6376 00000100:00080000:1.0:1787698475.941756:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698475.941757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941758:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941759:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698475.941761:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698475.941763:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941765:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941766:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698475.941768:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698475.941772:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698475.941775:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698475.941778:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698475.941783:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698475.941804:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.941805:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698475.941812:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698475.941813:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698475.941814:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698475.941817:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6376 00000100:00080000:1.0:1787698475.941819:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698475.941820:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941822:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941823:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698475.941825:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698475.941826:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941828:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941829:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698475.941830:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698475.941833:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698475.941837:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698475.941842:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698475.941847:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698475.941853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698475.941857:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698475.941860:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698475.941861:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698475.941863:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6376 00000100:00080000:1.0:1787698475.941866:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698475.941867:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941869:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698475.941870:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698475.941872:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698475.941874:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698475.941878:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698475.941878:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698475.941882:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698475.941885:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698475.941891:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698475.941897:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698475.941902:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698475.941907:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698475.941916:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6387) 00000100:00100000:2.0:1787698475.941928:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698475/real 1787698475] req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.941946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.941955:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698475.941963:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698475.941966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698475.941970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.941978:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126343296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698475.941986:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126343296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698475.941995:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698475.942012:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698475.942055:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942063:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698475.942076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698475.942083:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942094:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698475.942101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698475/real 1787698475] req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698475.942129:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698475.942132:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698475.942135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942143:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698475.942148:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942157:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698475.942169:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942176:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698475.942187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698475.942193:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942203:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698475.942209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698475/real 1787698475] req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942261:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698475.942270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698475.942273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698475.942277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698475.942291:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698475.942312:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942320:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698475.942331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698475.942338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942348:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698475.942354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698475/real 1787698475] req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942365:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942373:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698475.942380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698475.942384:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698475.942388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698475.942401:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698475.942441:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942449:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698475.942461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698475.942469:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698500 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942477:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698475.942482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698475/real 1787698475] req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698475.942501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698475.942508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698475.942511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698475.942515:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698475.942522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126343808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698475.942528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126343424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698475.942536:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698475.942542:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126343552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698475.942549:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698475.942554:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126343680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698475.942560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698475.942566:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126343808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698500 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698475.942573:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698476.453559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698476.453559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698476.453561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698476.581564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698477.477536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698477.477536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698477.477539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698477.605604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698477.861535:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301051247, at 4301051264 00000100:00000040:3.0:1787698477.861535:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301051247, at 4301051264 00000100:00100000:0.0:1787698478.501557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698478.501557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698478.501558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698478.629588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698479.525540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698479.525540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698479.525540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698479.653641:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698480.549516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698480.549536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698480.549566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698480.677594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698481.061557:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698481.061565:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698481.061569:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698481.061577:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6382 00000100:00080000:0.0:1787698481.061586:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698481.061588:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061590:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061593:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698481.061597:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698481.061599:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061600:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061602:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698481.061605:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061631:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698481.061656:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698481.061671:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698481.061693:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698481.061697:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698481.061699:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698481.061703:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6382 00000100:00080000:0.0:1787698481.061707:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698481.061710:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061713:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061715:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698481.061718:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698481.061720:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061722:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061723:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698481.061724:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698481.061733:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698481.061736:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698481.061746:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698481.061757:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698481.061759:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698481.061802:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698481.061804:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126343936:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698481.061818:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698481.061820:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698481.061824:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6382 00000100:00080000:0.0:1787698481.061826:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698481.061828:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061831:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061832:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698481.061834:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698481.061835:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061837:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061838:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698481.061840:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698481.061844:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698481.061848:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698481.061857:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698481.061858:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698481.061878:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698481.061913:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698481.061922:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:2.0:1787698481.061922:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698481.061927:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698481.061932:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6382 00000100:00080000:0.0:1787698481.061934:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698481.061936:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061938:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061944:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698481.061946:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698481.061948:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061950:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061951:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698481.061952:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698481.061955:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698481.061958:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698481.061963:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698481.061969:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698481.061976:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698481.061977:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698481.061979:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698481.061982:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698481.061985:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6382 00000100:00080000:0.0:1787698481.061988:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698481.061991:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.061992:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698481.061994:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698481.061997:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698481.062000:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698481.062007:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698481.062008:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698481.062010:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698481.062017:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698481.062025:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698481.062033:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698481.062042:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698481.062048:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698481.062056:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6392) 00000100:00100000:2.0:1787698481.062072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698481] req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062111:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698481.062122:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698481.062127:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698481.062131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126343936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698481.062180:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126343936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698481.062192:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698481.062209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698481.062229:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062235:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698481.062244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698481.062248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062256:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698481.062259:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698481] req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062272:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698481.062277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698481.062279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698481.062283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698481.062297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062305:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698481.062317:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062323:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698481.062333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698481.062339:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062349:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698481.062354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698481] req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062364:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062371:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698481.062378:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698481.062381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698481.062384:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698481.062403:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062410:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698481.062447:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062454:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698481.062465:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698481.062471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062480:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698481.062486:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698481] req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062496:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062503:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698481.062508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698481.062510:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698481.062513:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698481.062521:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062527:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698481.062535:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062539:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698481.062551:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698481.062555:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698506 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062561:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698481.062564:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698481/real 1787698481] req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062570:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698481.062575:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698481.062579:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698481.062581:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698481.062583:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698481.062586:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698481.062590:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126344064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698481.062599:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698481.062603:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126344192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698481.062607:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698481.062611:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126344320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698481.062615:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698481.062618:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126344448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698506 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698481.062621:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698481.573522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698481.573523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698481.573540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698481.701605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698482.597582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698482.597604:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698482.597708:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698482.725590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698482.981509:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301056367, at 4301056384 00000100:00000040:2.0:1787698482.981514:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301056367, at 4301056384 00000100:00100000:0.0:1787698483.621520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698483.621539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698483.621543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698483.749541:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698484.645570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698484.645594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698484.645613:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698484.773555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698485.669559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698485.669560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698485.669562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698485.797614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698486.181550:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698486.181566:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698486.181570:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698486.181583:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6387 00000100:00080000:0.0:1787698486.181595:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698486.181597:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181601:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181604:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698486.181609:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698486.181612:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181614:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181617:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698486.181619:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181635:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698486.181663:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698486.181676:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698486.181719:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698486.181722:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698486.181724:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698486.181727:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6387 00000100:00080000:0.0:1787698486.181731:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698486.181733:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181735:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181738:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698486.181740:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698486.181742:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181744:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181746:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698486.181748:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698486.181754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698486.181758:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698486.181764:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698486.181775:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126344576:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698486.181776:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698486.181786:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698486.181791:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698486.181793:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698486.181794:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698486.181797:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6387 00000100:00080000:0.0:1787698486.181799:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698486.181807:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181812:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181814:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787698486.181814:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698486.181817:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698486.181823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181829:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181833:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698486.181837:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698486.181840:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787698486.181841:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698486.181846:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698486.181857:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698486.181866:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698486.181873:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698486.181875:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698486.181876:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698486.181879:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6387 00000100:00080000:0.0:1787698486.181882:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698486.181884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181888:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698486.181890:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698486.181894:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698486.181896:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181897:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181901:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698486.181904:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698486.181908:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698486.181911:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698486.181917:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698486.181922:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698486.181928:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698486.181947:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698486.181949:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698486.181950:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698486.181953:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6387 00000100:00080000:0.0:1787698486.181956:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698486.181958:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181961:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181962:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698486.181964:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698486.181966:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698486.181968:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698486.181969:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698486.181972:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698486.181977:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698486/real 1787698486] req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698486.181978:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698486.181984:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698486.181992:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698486.181998:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182003:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698486.182009:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6397) 00000100:00000040:1.0:1787698486.182020:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698486.182031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698486.182035:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698486.182040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182049:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126344576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698486.182070:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126344576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698486.182081:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698486.182098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698486.182121:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182129:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698486.182140:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698486.182147:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182156:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698486.182162:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698486/real 1787698486] req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182185:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698486.182192:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698486.182195:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698486.182199:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698486.182212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698486.182230:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182237:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698486.182246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698486.182251:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182259:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698486.182274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698486/real 1787698486] req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182285:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698486.182301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698486.182304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698486.182308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698486.182320:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182328:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698486.182339:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182346:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698486.182356:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698486.182362:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182370:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698486.182375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698486/real 1787698486] req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698486.182400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698486.182403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698486.182406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126344960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698486.182456:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698486.182477:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182484:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698486.182493:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698486.182499:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698511 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182508:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698486.182513:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698486/real 1787698486] req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182523:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698486.182531:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698486.182539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698486.182542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698486.182545:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698486.182553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698486.182558:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126344704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698486.182565:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698486.182571:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126344832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698486.182577:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698486.182581:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126344960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698486.182588:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698486.182592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126345088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698511 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698486.182599:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698486.693560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698486.693560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698486.693560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698486.821590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698487.717501:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698487.717502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698487.717506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698487.845582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698488.101499:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301061487, at 4301061504 00000100:00000040:2.0:1787698488.101508:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301061487, at 4301061504 00000100:00100000:3.0:1787698488.741564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698488.741565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698488.741726:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698488.869589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698489.765547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698489.765583:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698489.765584:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698489.893648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698490.789523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698490.789523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698490.789523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698490.917619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698491.301588:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698491.301602:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698491.301609:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698491.301629:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6392 00000100:00080000:1.0:1787698491.301642:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698491.301646:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.301651:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.301655:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698491.301660:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698491.301663:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.301666:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.301669:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698491.301672:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698491.301691:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698491.301724:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698491.301742:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698491.301770:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698491.301774:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698491.301776:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698491.301781:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6392 00000100:00080000:1.0:1787698491.301785:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698491.301788:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.301792:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.301795:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698491.301798:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698491.301802:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.301806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.301811:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:2.0:1787698491.301812:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698491.301817:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698491.301832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126345216:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787698491.301833:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698491.301841:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698491.301854:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698491.301865:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698491.301868:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.301881:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698491.301889:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698491.301893:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698491.301895:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698491.301900:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6392 00000100:00080000:1.0:1787698491.301904:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698491.301907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.301911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.301913:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698491.301916:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698491.301920:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:2.0:1787698491.301923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698491.301927:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.301930:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698491.301934:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698491.301940:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698491.301941:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698491.301949:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698491.301974:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698491.301987:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698491/real 1787698491] req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698491.301998:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698491.302015:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698491.302019:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698491.302022:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698491.302027:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698491.302031:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698491.302035:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6392 00000100:00080000:2.0:1787698491.302037:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787698491.302043:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698491.302043:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698491.302053:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698491.302054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126345216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698491.302062:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698491.302065:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126345216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698491.302068:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698491.302073:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000020:00000040:2.0:1787698491.302077:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698491.302079:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.302084:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.302086:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698491.302089:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698491.302094:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698491.302096:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698491.302106:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698491.302107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345344:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698491.302124:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698491.302126:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302150:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698491.302160:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698491.302164:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698491.302172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698491.302177:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698491.302187:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698491.302191:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698491.302194:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:1.0:1787698491.302198:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6392 00000100:00100000:2.0:1787698491.302202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698491/real 1787698491] req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698491.302204:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698491.302213:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.302217:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.302220:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:2.0:1787698491.302222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698491.302224:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698491.302231:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698491.302237:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698491.302242:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698491.302249:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698491.302250:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698491.302257:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698491.302263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787698491.302266:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698491.302271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698491.302279:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698491.302281:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698491.302302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698491.302309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698491.302321:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302322:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698491.302339:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6402) 00000100:00000040:2.0:1787698491.302344:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302354:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698491.302367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698491.302372:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302382:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698491.302386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698491/real 1787698491] req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302398:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698491.302402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698491.302404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698491.302406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698491.302414:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126345344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698491.302444:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698491.302449:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126345472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698491.302453:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698491.302458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302464:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698491.302484:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302490:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698491.302502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698491.302508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302518:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698491.302524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698491/real 1787698491] req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302535:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302543:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698491.302550:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698491.302553:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698491.302556:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698491.302569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302577:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698491.302586:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302595:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698491.302603:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698491.302607:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698516 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302613:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698491.302617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698491/real 1787698491] req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698491.302645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698491.302653:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698491.302656:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698491.302660:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698491.302667:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698491.302671:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126345600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698491.302676:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698491.302680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126345728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698516 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698491.302684:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698491.813538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698491.813538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698491.813538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698491.941585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698492.838068:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698492.838292:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698492.838440:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698492.965607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698493.221508:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301066607, at 4301066624 00000100:00000040:3.0:1787698493.221508:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301066607, at 4301066624 00000100:00100000:2.0:1787698493.861758:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698493.861818:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698493.862186:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698493.989553:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698494.885538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698494.885538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698494.885538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698495.013713:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698495.909498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698495.909597:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698495.909617:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698496.037601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698496.421550:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698496.421560:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698496.421564:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698496.421576:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6397 00000100:00080000:0.0:1787698496.421592:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698496.421596:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421601:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421605:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698496.421610:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698496.421613:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421615:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421617:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698496.421620:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421634:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698496.421663:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698496.421676:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698496.421693:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698496.421695:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698496.421697:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698496.421700:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6397 00000100:00080000:0.0:1787698496.421704:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698496.421706:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421708:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421710:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698496.421712:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698496.421716:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421719:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421721:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698496.421725:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698496.421725:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698496.421739:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698496.421741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126345856:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698496.421747:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698496.421760:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698496.421766:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698496.421773:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698496.421773:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698496.421778:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698496.421781:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698496.421788:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6397 00000100:00000040:2.0:1787698496.421788:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698496.421793:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698496.421796:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421798:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421800:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698496.421801:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698496.421803:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421805:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421807:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698496.421809:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698496.421812:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698496.421816:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698496.421822:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698496.421827:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698496.421833:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698496.421838:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698496.421840:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698496.421841:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698496.421844:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6397 00000100:00080000:0.0:1787698496.421846:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698496.421848:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421850:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421851:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698496.421853:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698496.421855:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698496.421857:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421862:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421866:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698496.421869:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698496.421873:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698496.421888:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698496.421895:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698496.421900:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698496.421907:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698496.421909:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698496.421910:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698496.421913:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6397 00000100:00100000:2.0:1787698496.421917:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698496/real 1787698496] req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698496.421918:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698496.421923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698496.421928:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421931:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698496.421934:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698496.421941:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698496.421943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698496.421946:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698496.421950:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698496.421955:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698496.421959:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698496.421960:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698496.421968:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698496.421970:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698496.421974:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787698496.421977:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698496.421981:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698496.421993:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.421997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126345856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698496.422004:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6407) 00000100:00000040:2.0:1787698496.422007:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126345856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698496.422015:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698496.422029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422035:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698496.422045:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422049:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698496.422057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698496.422062:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422069:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698496.422072:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698496/real 1787698496] req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698496.422096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698496.422099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698496.422103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126345984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698496.422123:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698496.422144:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422152:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698496.422163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698496.422169:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422176:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698496.422179:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698496/real 1787698496] req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422186:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698496.422197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698496.422198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698496.422200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698496.422209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698496.422223:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422228:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698496.422237:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698496.422243:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422252:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698496.422258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698496/real 1787698496] req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698496.422292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698496.422299:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698496.422304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422311:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698496.422317:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698496.422334:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422339:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698496.422347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698496.422351:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698521 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422357:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698496.422362:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698496/real 1787698496] req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422373:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698496.422381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698496.422388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698496.422391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698496.422394:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698496.422401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698496.422407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126345984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698496.422416:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698496.422443:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126346112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698496.422451:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698496.422456:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126346240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698496.422463:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698496.422467:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126346368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698521 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698496.422474:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698496.933539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698496.933539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698496.933539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698497.061595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698497.957497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698497.957500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698497.957510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698498.085544:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698498.341509:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301071727, at 4301071744 00000100:00000040:3.0:1787698498.341509:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301071727, at 4301071744 00000100:00100000:1.0:1787698498.981506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698498.981506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698498.981506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698499.109613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698500.005546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698500.005546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698500.005546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698500.133634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698501.029508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698501.029508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698501.029509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698501.157642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698501.541528:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698501.541563:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698501.541567:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698501.541580:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6402 00000100:00080000:3.0:1787698501.541593:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698501.541597:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541602:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541606:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698501.541612:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698501.541615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541617:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541620:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698501.541623:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541635:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698501.541666:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698501.541681:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698501.541704:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698501.541708:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698501.541709:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698501.541713:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6402 00000100:00080000:3.0:1787698501.541717:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698501.541719:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541722:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541725:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698501.541728:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698501.541739:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541742:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541745:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698501.541750:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698501.541752:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698501.541761:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698501.541767:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698501.541771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126346496:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698501.541781:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698501.541804:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698501.541813:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698501.541817:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787698501.541820:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698501.541821:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698501.541829:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6402 00000100:00080000:3.0:1787698501.541835:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698501.541839:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541842:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541847:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541849:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698501.541851:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698501.541854:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541858:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541859:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698501.541862:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698501.541867:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698501.541872:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787698501.541884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698501.541885:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698501.541902:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.541910:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698501.541911:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698501.541922:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698501.541924:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698501.541928:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6402 00000100:00080000:3.0:1787698501.541933:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698501.541936:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541939:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541942:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698501.541944:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787698501.541949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698501/real 1787698501] req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698501.541949:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.541955:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.541959:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698501.541964:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698501.541972:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698501.541978:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698501.541984:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698501.541990:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698501.541998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698501.542007:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698501.542012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698501.542019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698501.542024:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698501.542024:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698501.542033:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698501.542037:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698501.542040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126346496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698501.542049:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126346496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698501.542051:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6402 00000100:00080000:3.0:1787698501.542058:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698501.542062:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698501.542063:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698501.542068:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698501.542072:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698501.542075:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698501.542078:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698501.542082:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698501.542085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698501.542086:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698501.542094:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698501.542103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346624:192.168.201.136@tcp:38:kworker.0 00000008:00100000:3.0:1787698501.542103:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698501.542112:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698501.542122:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698501.542128:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698501.542136:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698501.542145:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698501.542153:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698501.542163:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6412) 00000100:00000040:2.0:1787698501.542166:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542177:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698501.542184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698501/real 1787698501] req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542195:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698501.542210:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698501.542213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698501.542216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698501.542228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698501.542246:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542253:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698501.542261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698501.542267:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542275:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698501.542279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698501/real 1787698501] req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542294:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698501.542310:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698501.542312:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698501.542316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698501.542328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542336:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698501.542347:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542354:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698501.542362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698501.542367:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542376:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698501.542380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698501/real 1787698501] req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542403:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698501.542410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698501.542413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698501.542416:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126346880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698501.542450:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126346624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698501.542457:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698501.542463:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126346752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698501.542470:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698501.542474:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126346880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698501.542481:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698501.542487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698501.542507:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542513:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698501.542531:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698501.542537:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698526 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542545:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698501.542550:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698501/real 1787698501] req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698501.542568:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698501.542574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698501.542577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698501.542580:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698501.542587:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698501.542592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126347008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698526 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698501.542599:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698502.053539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698502.053539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698502.053539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698502.181628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698503.077496:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698503.077496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698503.077499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698503.205625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698503.461530:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301076847, at 4301076864 00000100:00000040:3.0:1787698503.461530:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301076847, at 4301076864 00000100:00100000:1.0:1787698504.101515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698504.101515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698504.101515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698504.229585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698505.125488:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698505.125488:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698505.125490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698505.253642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698506.149499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698506.149500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698506.149501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698506.277558:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698506.661533:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698506.661546:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698506.661551:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698506.661563:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6407 00000100:00080000:3.0:1787698506.661598:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698506.661601:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661605:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661608:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698506.661612:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698506.661614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661616:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661619:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698506.661621:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661634:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698506.661660:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698506.661672:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698506.661689:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698506.661692:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698506.661695:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698506.661699:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6407 00000100:00080000:3.0:1787698506.661716:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698506.661718:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661721:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661723:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698506.661726:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698506.661729:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661731:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661733:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698506.661736:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698506.661743:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698506.661752:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698506.661756:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698506.661757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126347136:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698506.661767:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698506.661776:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698506.661784:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698506.661787:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698506.661788:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698506.661791:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6407 00000100:00080000:3.0:1787698506.661794:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698506.661796:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698506.661797:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661800:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661802:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698506.661804:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698506.661805:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661807:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661809:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661812:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698506.661814:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698506.661818:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698506.661821:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698506.661827:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698506.661835:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698506.661844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698506.661852:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698506.661855:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698506.661856:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698506.661860:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6407 00000100:00000040:2.0:1787698506.661862:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698506.661863:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698506.661866:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661869:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661871:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698506.661873:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698506.661875:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661877:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661879:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698506.661881:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698506.661886:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698506.661890:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698506.661894:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698506/real 1787698506] req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698506.661897:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698506.661906:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.661910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698506.661913:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698506.661917:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698506.661918:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698506.661920:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698506.661921:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6407 00000100:00080000:3.0:1787698506.661924:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698506.661929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698506.661931:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698506.661933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698506.661933:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698506.661936:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698506.661937:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698506.661938:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698506.661941:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661943:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698506.661946:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126347136:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698506.661946:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698506.661948:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698506.661953:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698506.661956:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126347136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698506.661956:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698506.661963:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787698506.661966:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698506.661971:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698506.661980:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6417) 00000100:00000040:2.0:1787698506.661982:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.661991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698506.662005:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662012:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698506.662022:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698506.662026:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662034:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698506.662040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698506/real 1787698506] req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662050:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662059:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698506.662067:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698506.662071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698506.662075:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662087:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698506.662091:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698506.662106:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662110:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698506.662118:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698506.662122:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662128:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698506.662131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698506/real 1787698506] req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662143:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698506.662148:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698506.662149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698506.662151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698506.662159:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698506.662171:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662175:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698506.662180:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698506.662183:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662189:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698506.662192:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698506/real 1787698506] req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662198:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698506.662208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698506.662209:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698506.662212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662218:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698506.662223:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698506.662243:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662251:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698506.662261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698506.662268:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698531 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662286:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698506.662292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698506/real 1787698506] req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698506.662307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698506.662312:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698506.662314:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698506.662315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698506.662320:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698506.662323:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126347264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698506.662328:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698506.662333:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126347392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698506.662337:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698506.662340:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126347520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698506.662344:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698506.662346:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126347648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698531 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698506.662350:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698507.173555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698507.173555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698507.173555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698507.301601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698508.197496:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698508.197496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698508.197518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698508.325596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698508.581510:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301081967, at 4301081984 00000100:00000040:3.0:1787698508.581511:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301081967, at 4301081984 00000100:00100000:1.0:1787698509.221535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698509.221548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698509.221571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698509.349610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698510.245511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698510.245511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698510.245588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698510.373623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698511.269585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698511.269587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698511.269590:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698511.397609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698511.781560:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698511.781570:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698511.781574:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698511.781585:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6412 00000100:00080000:0.0:1787698511.781596:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698511.781598:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781602:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781605:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698511.781610:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698511.781628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781631:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781635:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698511.781639:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781658:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698511.781691:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698511.781704:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698511.781725:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698511.781731:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698511.781733:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698511.781737:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6412 00000100:00080000:0.0:1787698511.781740:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698511.781742:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781745:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781747:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698511.781749:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698511.781752:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781755:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781758:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698511.781760:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698511.781770:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698511.781771:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698511.781775:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698511.781789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126347776:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698511.781791:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698511.781806:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698511.781814:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698511.781816:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698511.781818:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698511.781820:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6412 00000100:00080000:0.0:1787698511.781823:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698511.781827:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698511.781828:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781834:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781838:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698511.781842:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698511.781846:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698511.781850:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781851:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781854:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698511.781856:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698511.781859:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698511.781863:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698511.781869:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698511.781880:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698511.781887:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698511.781889:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698511.781890:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698511.781892:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6412 00000100:00080000:0.0:1787698511.781895:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698511.781897:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781900:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781904:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781905:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698511.781907:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698511.781909:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781912:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698511.781914:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698511.781916:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698511.781920:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698511.781922:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698511.781929:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698511.781940:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698511.781946:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698511.781955:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698511.781960:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698511.781962:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698511.781967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698511/real 1787698511] req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698511.781970:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6412 00000100:00080000:0.0:1787698511.781978:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698511.781983:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.781987:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698511.781990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698511.781992:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698511.782001:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698511.782007:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:2.0:1787698511.782012:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698511.782013:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698511.782018:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698511.782021:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787698511.782022:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698511.782027:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698511.782029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698511.782032:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698511.782041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126347776:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698511.782042:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698511.782048:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126347776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698511.782057:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787698511.782058:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698511.782073:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782082:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698511.782090:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6422) 00000100:00100000:2.0:1787698511.782098:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698511.782117:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782127:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698511.782142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698511.782151:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782163:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698511.782170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698511/real 1787698511] req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782184:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782190:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698511.782196:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698511.782198:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698511.782200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126347904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698511.782209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698511.782224:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782229:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698511.782238:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698511.782242:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782249:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698511.782252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698511/real 1787698511] req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698511.782275:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698511.782276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698511.782287:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698511.782295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698511.782310:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782316:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698511.782322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698511.782327:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782333:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698511.782336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698511/real 1787698511] req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782348:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782354:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698511.782358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698511.782360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698511.782363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698511.782371:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126347904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698511.782376:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698511.782380:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126348032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698511.782384:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698511.782387:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126348160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698511.782392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698511.782397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698511.782412:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782417:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698511.782451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698511.782456:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698536 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782463:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698511.782466:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698511/real 1787698511] req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782477:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698511.782486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698511.782493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698511.782496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698511.782501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698511.782509:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698511.782515:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126348288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698536 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698511.782524:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698512.293503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698512.293504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698512.293508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698512.421651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698513.317547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698513.317547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698513.317547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698513.445608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698513.701512:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301087087, at 4301087104 00000100:00000040:3.0:1787698513.701512:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301087087, at 4301087104 00000100:00100000:1.0:1787698514.341515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698514.341515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698514.341516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698514.469665:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698515.365541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698515.365541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698515.365671:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698515.493643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698516.389527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698516.389527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698516.389527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698516.517639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698516.901589:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698516.901597:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698516.901601:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698516.901609:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6417 00000100:00080000:0.0:1787698516.901619:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698516.901622:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.901625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.901627:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698516.901632:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698516.901634:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.901636:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.901638:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698516.901641:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698516.901653:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698516.901676:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698516.901688:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.901713:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698516.901717:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698516.901719:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698516.901724:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6417 00000100:00080000:0.0:1787698516.901728:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698516.901733:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.901737:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.901740:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698516.901744:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698516.901747:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.901750:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.901753:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:2.0:1787698516.901761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.901762:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698516.901777:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698516.901794:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698516.901794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126348416:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698516.901817:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698516.901833:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.901843:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698516.901847:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.901852:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698516.901858:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698516.901866:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698516.901870:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6417 00000100:00080000:0.0:1787698516.901876:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698516.901881:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.901885:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.901887:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698516.901890:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698516.901893:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.901896:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.901899:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698516.901902:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698516.901908:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698516.901913:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787698516.901919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698516.901928:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698516.901940:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.901950:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698516.901954:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698516.901960:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698516.901961:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.901970:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6417 00000100:00080000:0.0:1787698516.901990:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698516.901995:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.901998:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.902001:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698516.902004:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698516.902008:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698516.902011:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.902014:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:2.0:1787698516.902021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698516/real 1787698516] req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.902022:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698516.902036:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698516.902044:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698516.902054:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698516.902058:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698516.902101:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698516.902103:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698516.902127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698516.902131:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698516.902135:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698516.902137:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698516.902142:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698516.902147:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698516.902152:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6417 00000100:00080000:0.0:1787698516.902161:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698516.902169:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698516.902175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126348416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698516.902181:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.902191:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698516.902196:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698516.902202:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698516.902204:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126348416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698516.902208:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698516.902220:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698516.902224:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698516.902228:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:0.0:1787698516.902229:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698516.902237:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698516.902245:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698516.902253:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698516.902265:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6428) 00000100:00100000:2.0:1787698516.902278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698516.902299:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902308:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698516.902325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698516.902334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902347:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698516.902354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698516/real 1787698516] req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698516.902386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698516.902390:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698516.902395:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902411:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698516.902442:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698516.902462:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902468:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698516.902477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698516.902483:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902490:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698516.902494:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698516/real 1787698516] req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902514:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698516.902524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698516.902542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698516.902547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698516.902563:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698516.902590:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902598:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698516.902612:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698516.902620:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902630:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698516.902637:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698516/real 1787698516] req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698516.902676:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698516.902680:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698516.902684:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698516.902699:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126348544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698516.902709:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698516.902717:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126348672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698516.902725:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698516.902732:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126348800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698516.902740:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698516.902750:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698516.902774:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902782:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698516.902796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698516.902804:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698541 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902814:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698516.902821:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698516/real 1787698516] req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698516.902844:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698516.902858:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698516.902862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698516.902866:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698516.902874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126348928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698516.902880:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126348928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698541 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698516.902889:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698517.413540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698517.413553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698517.413558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698517.541666:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698518.437533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698518.437542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698518.437543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698518.565703:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698518.821568:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301092207, at 4301092224 00000100:00000040:3.0:1787698518.821568:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301092207, at 4301092224 00000100:00100000:0.0:1787698519.461584:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698519.461584:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698519.461625:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698519.589618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698520.485520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698520.485520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698520.485524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698520.613648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698521.509509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698521.509528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698521.510488:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698521.637655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698522.021599:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698522.021614:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698522.021619:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698522.021634:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6423 00000100:00080000:3.0:1787698522.021649:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698522.021653:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.021685:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.021688:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698522.021693:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698522.021696:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.021698:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.021701:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698522.021703:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698522.021718:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698522.021752:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698522.021772:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698522.021802:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698522.021806:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698522.021808:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698522.021814:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6423 00000100:00080000:3.0:1787698522.021819:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698522.021822:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.021825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.021828:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698522.021832:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698522.021840:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.021844:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.021847:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698522.021851:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698522.021860:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698522.021862:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698522.021881:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126349056:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698522.021882:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698522.021899:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698522.021912:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698522.021917:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698522.021936:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698522.021938:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698522.021945:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698522.021947:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698522.021953:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6423 00000100:00080000:3.0:1787698522.021976:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698522.021979:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.021982:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.021985:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698522.021988:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698522.021992:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.021996:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.021999:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698522.022003:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698522.022009:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698522.022015:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698522.022027:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698522.022048:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698522.022049:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698522.022084:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698522.022087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349184:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698522.022099:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698522.022102:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698522.022107:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6423 00000100:00000040:2.0:1787698522.022112:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698522.022114:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698522.022123:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698522.022128:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698522.022132:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.022136:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698522.022138:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698522.022143:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.022147:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.022150:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698522.022153:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698522.022155:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000008:00100000:3.0:1787698522.022161:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698522.022167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698522.022169:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698522.022180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349312:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787698522.022187:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698522.022199:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698522.022205:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698522.022214:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698522.022226:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698522.022227:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00080000:3.0:1787698522.022233:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698522.022237:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698522.022240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698522.022244:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6423 00000100:00080000:3.0:1787698522.022253:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698522.022257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787698522.022261:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.022267:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698522.022272:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698522.022279:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698522.022285:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698522.022287:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698522.022300:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787698522.022317:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698522.022322:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698522.022325:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698522.022329:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698522.022335:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698522.022339:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698522.022349:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698522.022361:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698522.022384:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6433) 00000100:00000040:2.0:1787698522.022469:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698522.022482:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698522.022506:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698522.022516:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698522.022534:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698522.533627:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698522.533979:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698522.534211:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698522.661630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698523.557563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698523.557563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698523.557563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698523.685588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698523.941532:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301097327, at 4301097344 00000100:00000040:3.0:1787698523.941532:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301097327, at 4301097344 00000100:00100000:1.0:1787698524.581554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698524.581554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698524.581554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698524.709634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787698525.093690:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787698525.093729:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787698525.093742:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698525.093772:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698525.093787:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698525.093799:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698525.093812:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698525.093827:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698525.093842:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698525.093856:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.093870:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698525] req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698525.093884:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698525.093903:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698547 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.093942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698525.093959:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698525.093975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698525.093981:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698525.093987:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.093997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349312:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787698525.094009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698525] req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094024:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698525.094035:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698525.094044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698525.094048:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698525.094053:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094062:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349184:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787698525.094070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698525] req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094086:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698525.094095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698525.094104:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698525.094108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698525.094113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094122:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126349056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698525.094131:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126349312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698525.094149:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698525.094162:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126349184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698525.094170:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698525.094176:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126349056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698525.094184:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698525.094194:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698525] req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094209:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698525.094218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698525.094226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698525.094231:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698525.094235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094243:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349568:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787698525.094251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698522/real 1787698525] req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094261:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698525.094269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698525.094277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698525.094280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698525.094285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698525.094292:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698525.094299:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126349568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698525.094307:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698525.094313:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126349440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698547 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698525.094322:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698525.605526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698525.605530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698525.605533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698525.733564:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698526.629546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698526.629546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698526.629546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698526.757615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698527.141586:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698527.141601:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698527.141607:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698527.141623:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6428 00000100:00080000:1.0:1787698527.141639:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698527.141643:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.141648:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.141653:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698527.141660:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698527.141664:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.141667:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.141671:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698527.141675:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698527.141691:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698527.141728:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698527.141746:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.141774:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698527.141779:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698527.141781:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698527.141787:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6428 00000100:00080000:1.0:1787698527.141791:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698527.141795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.141799:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.141802:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698527.141806:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698527.141810:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.141813:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.141816:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698527.141820:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698527.141834:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698527.141836:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698527.141841:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698527.141861:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126349696:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698527.141866:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698527.141880:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.141892:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698527.141896:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698527.141899:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698527.141904:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6428 00000100:00000040:2.0:1787698527.141909:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.141911:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698527.141920:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.141927:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698527.141932:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698527.141933:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698527.141939:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698527.141943:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.141946:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.141950:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698527.141954:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698527.141960:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698527.141966:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698527.141977:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698527.141989:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698527.141993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698527.142012:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698527.142017:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698527.142023:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698527.142023:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.142033:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6428 00000100:00080000:1.0:1787698527.142041:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698527.142046:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.142050:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.142052:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698527.142055:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698527.142058:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.142061:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.142064:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698527.142067:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698527.142072:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698527.142078:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698527.142090:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698527.142098:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698527.142109:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.142124:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698527.142129:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787698527.142131:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698527.142132:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698527.142143:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6428 00000100:00080000:1.0:1787698527.142149:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698527.142156:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698527.142159:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.142167:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698527.142173:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698527.142179:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:2.0:1787698527.142182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:1.0:1787698527.142184:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698527.142191:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698527.142196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787698527.142197:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698527.142204:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698527.142207:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698527.142211:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698527.142221:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698527.142228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126349696:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698527.142237:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698527.142242:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126349696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698527.142265:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698527.142269:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698527.142284:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6438) 00000100:00000040:2.0:1787698527.142298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698527.142337:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142346:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698527.142376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698527.142385:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142396:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698527.142403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142456:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698527.142478:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698527.142483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698527.142487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698527.142501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698527.142525:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142534:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698527.142550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698527.142558:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142570:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698527.142577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142600:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698527.142610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698527.142614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698527.142618:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126349952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698527.142633:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142643:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698527.142659:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142667:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698527.142680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698527.142688:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142700:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698527.142706:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142721:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142730:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698527.142738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698527.142742:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698527.142746:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142753:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698527.142759:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142767:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698527.142781:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142789:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698527.142800:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698527.142807:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698552 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142817:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698527.142823:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698527/real 1787698527] req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698527.142844:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698527.142853:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698527.142857:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698527.142861:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698527.142868:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698527.142875:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126349824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698527.142883:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698527.142890:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126349952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698527.142897:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698527.142902:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126350080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698527.142910:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698527.142915:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126350208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698552 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698527.142922:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698527.653510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698527.653531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698527.653621:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698527.781647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698528.677555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698528.677555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698528.677604:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698528.805585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698529.061552:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301102447, at 4301102464 00000100:00000040:3.0:1787698529.061552:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301102447, at 4301102464 00000100:00100000:1.0:1787698529.701549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698529.701549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698529.701552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698529.829618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698530.725526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698530.725527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698530.725571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698530.853596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698531.749552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698531.749552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698531.749729:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698531.877686:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698532.261598:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698532.261612:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698532.261619:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698532.261656:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6433 00000100:00080000:3.0:1787698532.261670:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698532.261675:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698532.261681:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.261685:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698532.261691:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698532.261696:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698532.261699:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.261703:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698532.261707:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698532.261723:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698532.261758:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698532.261777:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698532.261805:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698532.261819:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698532.261821:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698532.261826:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6433 00000100:00080000:3.0:1787698532.261830:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698532.261835:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698532.261840:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.261843:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698532.261847:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698532.261851:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698532.261858:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698532.261860:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698532.261873:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698532.261881:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698532.261885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126350336:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698532.261897:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698532.261907:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698532.261925:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698532.261927:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.261957:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698532.261968:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698532.261979:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698532.261983:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698532.261985:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698532.261990:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6433 00000800:00000100:2.0:1787698532.262003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698532.262006:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698532.262012:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698532.262016:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.262019:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698532.262023:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698532.262029:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698532.262030:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698532.262038:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.262044:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698532.262050:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698532.262056:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698532.262062:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698532.262074:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698532.262080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698532/real 1787698532] req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262122:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698532.262128:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698532.262146:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698532.262154:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698532.262165:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698532.262166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698532.262172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698532.262178:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6433 00000100:00000040:2.0:1787698532.262182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698532.262186:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698532.262195:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698532.262201:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126350336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698532.262203:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.262210:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698532.262213:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698532.262215:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126350336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698532.262219:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698532.262228:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.262235:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000020:00000040:2.0:1787698532.262239:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698532.262240:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698532.262250:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698532.262256:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698532.262262:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262288:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350464:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787698532.262297:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698532.262309:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262316:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698532.262323:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698532.262333:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698532.262338:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698532.262341:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698532.262352:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6433 00000800:00000100:2.0:1787698532.262359:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698532.262359:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698532.262367:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698532.262374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698532.262374:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.262381:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698532.262386:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698532.262393:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698532.262395:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787698532.262397:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698532.262402:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787698532.262406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698532/real 1787698532] req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698532.262409:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698532.262448:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698532.262454:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698532.262467:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698532.262476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698532.262487:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698532.262504:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6443) 00000100:00080000:2.0:1787698532.262522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698532.262527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698532.262533:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698532.262550:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698532.262575:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262584:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698532.262596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698532.262604:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262616:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698532.262628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698532/real 1787698532] req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262660:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698532.262669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698532.262673:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698532.262678:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698532.262695:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126350464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698532.262705:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698532.262712:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126350592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698532.262721:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698532.262729:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262740:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698532.262754:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262763:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698532.262776:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698532.262783:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262794:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698532.262800:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698532/real 1787698532] req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262824:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698532.262832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698532.262836:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698532.262840:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262849:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698532.262855:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262865:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698532.262878:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262886:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698532.262899:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698532.262907:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698557 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262920:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698532.262925:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698532/real 1787698532] req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262933:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698532.262939:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698532.262944:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698532.262946:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698532.262948:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698532.262953:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126350848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698532.262957:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126350720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698532.262962:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698532.262967:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126350848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698557 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698532.262971:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698532.773544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698532.773544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698532.773548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698532.901600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698533.797570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698533.797570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698533.797575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698533.925559:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698534.181527:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301107567, at 4301107584 00000100:00000040:3.0:1787698534.181527:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301107567, at 4301107584 00000100:00100000:3.0:1787698534.821525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698534.821526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698534.821526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698534.949672:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698535.845585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698535.845585:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698535.845592:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698535.973623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698536.869567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698536.869578:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698536.869587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698536.997636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698537.381557:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698537.381568:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698537.381573:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698537.381584:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6438 00000100:00080000:3.0:1787698537.381619:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698537.381623:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381627:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381630:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698537.381635:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698537.381637:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381639:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381642:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698537.381651:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381665:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698537.381696:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698537.381710:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698537.381732:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698537.381735:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698537.381737:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698537.381740:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6438 00000100:00080000:3.0:1787698537.381743:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698537.381744:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381747:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381750:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698537.381752:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698537.381755:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381759:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381761:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698537.381763:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698537.381777:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698537.381783:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698537.381783:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698537.381801:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698537.381817:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.381818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126350976:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698537.381844:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698537.381846:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698537.381847:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698537.381851:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6438 00000100:00080000:3.0:1787698537.381854:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698537.381858:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381860:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381862:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698537.381864:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698537.381866:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381868:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381874:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787698537.381875:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698537.381881:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698537.381890:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698537.381900:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698537.381906:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698537.381913:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698537.381922:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698537.381938:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698537.381941:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698537.381942:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698537.381948:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6438 00000800:00000100:2.0:1787698537.381952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698537.381955:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698537.381960:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381965:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698537.381968:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698537.381972:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.381975:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.381978:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698537.381980:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698537.381981:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698537.381991:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698537.381999:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698537.382009:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698537.382016:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698537.382024:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698537.382027:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698537.382032:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698537.382034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698537/real 1787698537] req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698537.382037:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6438 00000100:00080000:3.0:1787698537.382044:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698537.382055:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.382058:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.382061:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:2.0:1787698537.382064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698537.382064:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698537.382075:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698537.382085:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698537.382094:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698537.382103:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698537.382114:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698537.382123:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698537.382123:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698537.382139:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698537.382141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698537.382156:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698537.382159:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698537.382188:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698537.382190:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6448) 00000100:00100000:2.0:1787698537.382204:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126350976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698537.382213:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126350976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698537.382226:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698537.382245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698537.382274:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382282:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698537.382294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698537.382301:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382311:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698537.382316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698537/real 1787698537] req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698537.382342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698537.382345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698537.382349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382355:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698537.382361:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382375:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698537.382386:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382392:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698537.382402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698537.382407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382415:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698537.382442:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698537/real 1787698537] req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698537.382468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698537.382471:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698537.382474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382481:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698537.382486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698537.382505:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382512:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698537.382521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698537.382527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382535:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698537.382540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698537/real 1787698537] req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382558:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698537.382564:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698537.382567:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698537.382570:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382577:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698537.382583:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382589:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698537.382598:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382609:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698537.382618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698537.382624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698562 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382632:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698537.382638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698537/real 1787698537] req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698537.382656:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698537.382662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698537.382665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698537.382668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698537.382675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698537.382680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126351104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698537.382687:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698537.382693:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126351232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698537.382699:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698537.382704:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126351360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698537.382711:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698537.382715:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126351488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698562 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698537.382721:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698537.893536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698537.893537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698537.893554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698538.021634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698538.917591:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698538.917593:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698538.917603:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698539.045621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698539.301489:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301112687, at 4301112704 00000100:00000040:3.0:1787698539.301551:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301112687, at 4301112704 00000100:00100000:0.0:1787698539.941536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698539.941536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698539.941555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698540.069649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698540.965570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698540.965570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698540.965571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698541.093597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698541.989538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698541.989546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698541.989549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698542.117644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698542.501565:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698542.501576:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698542.501581:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698542.501598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6443 00000100:00080000:0.0:1787698542.501610:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698542.501613:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.501616:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.501620:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698542.501625:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698542.501627:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.501629:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.501632:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698542.501634:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698542.501650:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698542.501685:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698542.501704:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698542.501733:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698542.501736:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698542.501738:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698542.501742:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6443 00000100:00080000:0.0:1787698542.501745:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698542.501747:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.501749:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.501751:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698542.501754:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698542.501756:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.501757:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.501759:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698542.501761:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698542.501770:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698542.501782:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698542.501794:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698542.501803:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698542.501812:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698542.501815:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698542.501818:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698542.501827:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6443 00000100:00000040:2.0:1787698542.501829:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698542.501831:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698542.501839:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.501854:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.501863:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698542.501868:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698542.501871:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.501876:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698542.501877:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126351616:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698542.501886:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698542.501890:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698542.501894:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698542.501899:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698542.501906:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698542.501921:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.501922:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698542.502002:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698542.502010:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698542.502014:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698542.502017:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698542.502025:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6443 00000100:00080000:0.0:1787698542.502031:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698542.502035:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.502039:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.502042:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698542.502045:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:2.0:1787698542.502046:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698542.502054:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698542.502059:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.502062:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698542.502068:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698542.502069:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698542.502107:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698542.502118:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698542.502123:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698542/real 1787698542] req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698542.502135:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698542.502144:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698542.502165:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502178:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698542.502181:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698542.502193:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698542.502198:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698542.502213:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6443 00000100:00080000:2.0:1787698542.502214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698542.502225:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698542.502225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698542.502241:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698542.502242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698542.502258:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698542.502269:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:2.0:1787698542.502271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126351616:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698542.502282:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698542.502283:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126351616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698542.502301:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698542.502304:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698542.502323:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698542.502324:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698542.502333:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787698542.502344:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351744:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698542.502346:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698542.502357:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698542.502361:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502369:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698542.502370:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:2.0:1787698542.502387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698542.502390:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698542.502398:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698542.502472:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502472:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698542.502490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698542/real 1787698542] req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698542.502497:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6453) 00000100:00100000:2.0:1787698542.502504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502515:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698542.502523:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698542.502526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698542.502528:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698542.502537:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502542:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698542.502550:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502554:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698542.502560:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698542.502564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502570:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698542.502573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698542/real 1787698542] req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698542.502589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698542.502591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698542.502593:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502597:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126351872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698542.502601:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698542.502612:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502616:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698542.502621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698542.502624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502630:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698542.502633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698542/real 1787698542] req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502643:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698542.502647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698542.502649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698542.502651:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698542.502659:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126351744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698542.502663:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698542.502666:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126351872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698542.502670:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698542.502673:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126352000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698542.502677:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698542.502682:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502687:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698542.502693:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502697:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698542.502702:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698542.502706:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698567 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502711:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698542.502731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698542/real 1787698542] req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502738:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698542.502743:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698542.502749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698542.502751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698542.502754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698542.502759:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698542.502764:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126352128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698567 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698542.502770:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698543.013595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698543.013602:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698543.013603:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698543.141533:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698544.037552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698544.037555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698544.037557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698544.165631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698544.421555:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301117807, at 4301117824 00000100:00000040:2.0:1787698544.421556:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301117807, at 4301117824 00000100:00100000:1.0:1787698545.061512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698545.061516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698545.061547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698545.189531:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698546.085516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698546.085516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698546.085554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698546.213577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698547.109517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698547.109518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698547.109518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698547.237613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698547.621557:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698547.621565:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698547.621568:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698547.621576:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6448 00000100:00080000:0.0:1787698547.621586:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698547.621594:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621600:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621602:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698547.621606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698547.621609:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621611:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621613:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698547.621615:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621627:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698547.621653:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698547.621667:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698547.621689:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698547.621693:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698547.621695:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698547.621698:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6448 00000100:00080000:0.0:1787698547.621701:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698547.621703:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621705:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621706:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698547.621709:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698547.621712:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621715:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621718:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698547.621721:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698547.621730:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698547.621733:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698547.621736:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698547.621749:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698547.621750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126352256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698547.621763:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698547.621771:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698547.621773:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698547.621774:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698547.621777:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6448 00000100:00000040:2.0:1787698547.621780:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698547.621782:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698547.621788:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621793:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621797:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698547.621800:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698547.621802:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621803:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621804:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698547.621806:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698547.621813:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787698547.621814:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698547.621819:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698547.621829:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698547.621838:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698547.621848:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698547.621850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698547.621854:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698547.621857:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698547.621863:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6448 00000100:00080000:0.0:1787698547.621867:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698547.621871:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698547.621873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621880:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621884:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698547.621887:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698547.621889:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621893:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698547.621895:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698547.621899:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698547.621902:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698547.621911:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698547.621914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698547/real 1787698547] req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698547.621925:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698547.621934:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698547.621936:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698547.621937:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698547.621942:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6448 00000100:00100000:2.0:1787698547.621943:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698547.621945:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698547.621950:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621955:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698547.621958:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698547.621960:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698547.621961:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698547.621968:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698547.621974:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698547.621976:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698547.621977:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698547.621982:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787698547.621984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:0.0:1787698547.621986:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698547.621989:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698547.621991:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698547.621999:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698547.622012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126352256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698547.622024:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126352256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698547.622027:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698547.622039:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698547.622041:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6458) 00000100:00000040:2.0:1787698547.622055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622062:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698547.622073:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622081:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698547.622089:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698547.622093:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622099:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698547.622103:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698547/real 1787698547] req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622110:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622115:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698547.622119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698547.622121:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698547.622123:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698547.622130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698547.622141:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622145:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698547.622150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698547.622153:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622158:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698547.622161:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698547/real 1787698547] req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622167:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698547.622175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698547.622177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698547.622179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698547.622186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698547.622197:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622201:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698547.622205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698547.622208:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622216:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698547.622219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698547/real 1787698547] req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698547.622233:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698547.622235:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698547.622237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698547.622243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698547.622254:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622258:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698547.622262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698547.622265:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698572 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622270:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698547.622273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698547/real 1787698547] req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698547.622284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698547.622288:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698547.622289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698547.622291:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698547.622295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126352768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698547.622298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126352384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698547.622302:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698547.622305:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126352512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698547.622309:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698547.622311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126352640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698547.622315:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698547.622317:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126352768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698572 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698547.622321:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698548.133537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698548.133537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698548.133537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698548.261577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698549.157513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698549.157513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698549.157562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698549.285633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698549.541538:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301122927, at 4301122944 00000100:00000040:3.0:1787698549.541538:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301122927, at 4301122944 00000100:00100000:1.0:1787698550.181510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698550.181510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698550.181688:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698550.309648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698551.205895:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698551.209017:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698551.209091:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698551.333606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698552.229558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698552.229558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698552.229558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698552.357535:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698552.741601:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698552.741620:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698552.741625:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698552.741639:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6453 00000100:00080000:1.0:1787698552.741653:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698552.741656:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.741661:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.741664:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698552.741670:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698552.741673:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.741675:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.741678:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698552.741682:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698552.741698:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698552.741730:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698552.741746:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698552.741773:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698552.741777:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698552.741779:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698552.741783:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6453 00000100:00080000:1.0:1787698552.741787:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698552.741790:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.741793:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.741796:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698552.741798:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698552.741803:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.741806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.741808:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698552.741813:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698552.741814:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698552.741826:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698552.741829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126352896:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698552.741834:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698552.741848:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698552.741858:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.741863:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698552.741876:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698552.741880:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698552.741882:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698552.741894:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6453 00000100:00000040:2.0:1787698552.741894:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698552.741901:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698552.741905:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.741908:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.741910:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698552.741924:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698552.741928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.741930:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.741933:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698552.741936:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698552.741941:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698552.741947:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787698552.741949:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698552.741961:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698552.741969:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698552.741974:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698552.741991:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698552.741994:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698552.741996:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698552.742002:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6453 00000100:00100000:2.0:1787698552.742004:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698552/real 1787698552] req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698552.742007:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698552.742015:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.742019:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.742021:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698552.742024:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787698552.742025:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698552.742030:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.742034:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.742036:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698552.742042:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698552.742049:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698552.742049:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698552.742057:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698552.742064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698552.742070:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698552.742072:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698552.742080:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698552.742096:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126352896:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698552.742113:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698552.742117:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126352896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698552.742118:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698552.742122:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698552.742129:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6453 00000020:00000040:2.0:1787698552.742130:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787698552.742133:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698552.742138:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.742141:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698552.742143:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698552.742145:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698552.742148:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698552.742153:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698552.742153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698552.742156:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698552.742161:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698552.742170:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698552.742174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353024:192.168.201.136@tcp:38:kworker.0 00000020:00000040:1.0:1787698552.742176:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698552.742189:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698552.742196:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698552.742203:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742211:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698552.742221:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6463) 00000800:00000100:2.0:1787698552.742236:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698552.742248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742259:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698552.742266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698552/real 1787698552] req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698552.742294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698552.742298:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698552.742302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698552.742315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742324:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698552.742336:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742341:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698552.742349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698552.742354:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742360:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698552.742363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698552/real 1787698552] req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698552.742380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698552.742382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698552.742384:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742392:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698552.742396:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742401:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698552.742408:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742412:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698552.742417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698552.742439:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742445:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698552.742449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698552/real 1787698552] req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698552.742465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698552.742467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698552.742469:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742473:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698552.742477:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126353024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698552.742482:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698552.742486:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126353152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698552.742491:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698552.742493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126353280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698552.742497:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698552.742502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742506:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698552.742513:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742518:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698552.742524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698552.742527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698577 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742532:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698552.742535:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698552/real 1787698552] req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698552.742547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698552.742552:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698552.742554:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698552.742556:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698552.742560:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698552.742562:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126353408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698577 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698552.742566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698553.253547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698553.253548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698553.253835:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698553.384227:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698554.277561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698554.277613:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698554.277719:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698554.405591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698554.661499:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301128047, at 4301128064 00000100:00000040:2.0:1787698554.661516:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301128047, at 4301128064 00000100:00100000:0.0:1787698555.301517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698555.301517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698555.301556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698555.429625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698556.325562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698556.325562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698556.325608:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698556.453616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698557.349490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698557.349490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698557.349493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698557.477631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698557.861539:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698557.861553:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698557.861558:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698557.861572:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6458 00000100:00080000:3.0:1787698557.861584:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698557.861589:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698557.861593:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.861597:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698557.861602:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698557.861605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698557.861608:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.861611:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698557.861615:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698557.861632:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698557.861666:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698557.861684:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698557.861710:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698557.861714:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698557.861716:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698557.861721:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6458 00000100:00080000:3.0:1787698557.861725:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698557.861748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698557.861756:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698557.861764:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.861767:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698557.861771:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00100000:2.0:1787698557.861772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126353536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698557.861778:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698557.861783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.861786:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698557.861788:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698557.861800:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698557.861806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698557.861813:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698557.861819:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698557.861846:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698557.861856:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698557.861866:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698557.861869:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698557.861871:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698557.861876:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6458 00000100:00080000:3.0:1787698557.861881:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698557.861883:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:2.0:1787698557.861887:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698557.861890:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.861894:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698557.861897:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698557.861901:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698557.861904:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698557.861907:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698557.861909:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698557.861921:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698557.861927:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698557.861933:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698557.861936:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698557/real 1787698557] req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698557.861947:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698557.861953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698557.861969:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.861970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698557.861998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698557.861998:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698557.862006:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698557.862008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698557.862009:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698557.862017:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698557.862018:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6458 00000100:00080000:3.0:1787698557.862026:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698557.862031:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126353536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698557.862034:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698557.862040:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.862044:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:2.0:1787698557.862045:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126353536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698557.862049:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698557.862057:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698557.862063:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698557.862063:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.862067:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698557.862071:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698557.862077:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698557.862082:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698557.862085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698557.862095:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698557.862096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698557.862114:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862118:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862141:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698557.862149:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698557.862153:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698557.862155:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787698557.862158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698557.862162:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6458 00000100:00080000:3.0:1787698557.862168:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698557.862170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698557.862174:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698557.862181:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.862187:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698557.862191:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:3.0:1787698557.862191:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698557.862197:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698557.862200:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698557/real 1787698557] req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698557.862205:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698557.862211:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787698557.862216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698557.862217:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698557.862227:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698557.862229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698557.862237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787698557.862237:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698557.862244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698557.862247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353664:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787698557.862255:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698557.862259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698557.862288:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862289:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862318:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698557.862322:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6468) 00000800:00000100:2.0:1787698557.862330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698557.862335:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862342:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698557.862346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698557/real 1787698557] req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698557.862368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698557.862370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698557.862372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698557.862380:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126353664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698557.862385:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698557.862388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126353792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698557.862392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698557.862397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698557.862411:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862415:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698557.862438:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698557.862442:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862448:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698557.862451:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698557/real 1787698557] req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862457:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862462:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698557.862466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698557.862468:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698557.862470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862474:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126353920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698557.862477:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862481:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698557.862488:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862492:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698557.862497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698557.862500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698582 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862508:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698557.862511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698557/real 1787698557] req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698557.862522:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698557.862525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698557.862527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698557.862529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698557.862533:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698557.862535:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126353920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698557.862540:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698557.862543:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126354048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698582 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698557.862548:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698558.373547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698558.373547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698558.373547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698558.501626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698559.397516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698559.397516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698559.397542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698559.525654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698559.781550:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301133167, at 4301133184 00000100:00000040:3.0:1787698559.781550:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301133167, at 4301133184 00000100:00100000:3.0:1787698560.421581:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698560.421855:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698560.422118:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698560.549587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698561.445556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698561.445556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698561.445561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698561.573614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698562.469542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698562.469542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698562.469542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698562.597583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698562.981572:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698562.981581:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698562.981585:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698562.981596:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6463 00000100:00080000:2.0:1787698562.981606:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698562.981609:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981612:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981614:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698562.981619:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698562.981621:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981623:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981625:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698562.981628:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981649:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698562.981677:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698562.981691:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698562.981724:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698562.981728:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698562.981730:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698562.981735:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6463 00000100:00080000:2.0:1787698562.981753:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698562.981757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981760:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981763:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698562.981765:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698562.981770:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698562.981771:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981777:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981782:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698562.981788:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698562.981800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126354176:192.168.201.136@tcp:250:kworker.0 00000008:00100000:2.0:1787698562.981806:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698562.981813:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698562.981825:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698562.981836:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698562.981844:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698562.981845:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698562.981849:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:1.0:1787698562.981857:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698562.981859:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698562.981864:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6463 00000100:00080000:2.0:1787698562.981867:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698562.981869:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981871:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981873:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698562.981875:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698562.981877:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981879:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981880:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698562.981883:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698562.981887:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698562.981890:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698562.981895:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698562.981899:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698562.981907:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698562.981912:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698562.981915:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698562.981916:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698562.981918:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698562.981925:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6463 00000100:00080000:2.0:1787698562.981934:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698562.981936:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981938:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981940:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698562.981941:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698562.981944:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.981946:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698562.981947:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00100000:1.0:1787698562.981950:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698562/real 1787698562] req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698562.981952:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698562.981957:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698562.981960:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698562.981969:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698562.981976:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.981983:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698562.981984:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698562.981989:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698562.981991:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698562.982000:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6463 00000100:00080000:2.0:1787698562.982004:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698562.982005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698562.982010:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.982017:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698562.982018:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:2.0:1787698562.982020:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:1.0:1787698562.982024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:2.0:1787698562.982025:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698562.982029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698562.982032:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698562.982039:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698562.982042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126354176:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787698562.982043:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698562.982049:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698562.982052:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126354176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698562.982056:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698562.982061:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:2.0:1787698562.982062:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698562.982075:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698562.982079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698562.982101:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982107:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698562.982115:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6474) 00000100:00000040:1.0:1787698562.982123:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982129:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698562.982138:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698562.982144:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982152:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698562.982156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698562/real 1787698562] req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982163:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982168:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698562.982173:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698562.982174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698562.982176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698562.982184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982189:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698562.982197:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982202:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698562.982207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698562.982213:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982221:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698562.982226:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698562/real 1787698562] req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982237:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698562.982255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698562.982257:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698562.982260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698562.982268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698562.982283:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982288:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698562.982295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698562.982299:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982307:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698562.982312:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698562/real 1787698562] req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982323:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982331:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698562.982338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698562.982342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698562.982346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698562.982357:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126354304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698562.982362:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698562.982366:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126354432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698562.982371:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698562.982375:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126354560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698562.982379:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698562.982384:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982390:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698562.982398:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982403:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698562.982410:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698562.982414:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698587 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982437:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698562.982440:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698562/real 1787698562] req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698562.982453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698562.982457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698562.982459:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698562.982461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698562.982465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698562.982470:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126354688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698587 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698562.982474:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698563.493536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698563.493542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698563.493542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698563.621618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698564.517528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698564.517529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698564.517622:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698564.645583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698564.901521:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301138287, at 4301138304 00000100:00000040:3.0:1787698564.901521:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301138287, at 4301138304 00000100:00100000:0.0:1787698565.541549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698565.541549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698565.541550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698565.669583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698566.565526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698566.565526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698566.565526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698566.693663:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698567.589552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698567.589552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698567.589561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698567.717581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698568.101520:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698568.101530:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698568.101534:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698568.101545:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6469 00000100:00080000:1.0:1787698568.101556:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698568.101558:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101562:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101564:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698568.101569:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698568.101573:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101575:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101579:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698568.101582:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101599:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698568.101630:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698568.101647:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.101680:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698568.101685:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698568.101686:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698568.101691:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6469 00000100:00080000:1.0:1787698568.101696:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698568.101700:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101703:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101706:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698568.101709:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698568.101714:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698568.101715:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101724:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698568.101729:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698568.101737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126354816:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787698568.101739:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698568.101744:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698568.101755:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698568.101762:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.101768:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.101770:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698568.101777:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698568.101780:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698568.101787:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6469 00000100:00080000:1.0:1787698568.101792:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698568.101796:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101797:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101800:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101801:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698568.101803:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698568.101805:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101808:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698568.101809:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698568.101812:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698568.101815:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698568.101820:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698568.101825:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.101831:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787698568.101835:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698568.101836:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698568.101839:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698568.101843:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6469 00000100:00080000:1.0:1787698568.101845:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698568.101847:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101851:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698568.101853:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698568.101856:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698568.101858:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101867:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698568.101871:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698568.101876:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698568.101879:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698568.101885:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698568.101891:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.101897:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698568.101898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698568.101904:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698568.101907:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698568.101913:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6469 00000100:00080000:1.0:1787698568.101916:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698568.101918:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698568.101923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698568.101924:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698568.101927:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698568.101932:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698568.101937:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698568.101939:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698568.101943:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698568.101944:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698568.101946:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698568.101951:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698568.101954:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698568.101958:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698568.101959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698568.101967:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698568.101972:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.101986:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126354816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698568.101988:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6479) 00000100:00000040:2.0:1787698568.101994:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126354816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698568.102002:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698568.102014:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102020:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698568.102037:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102046:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698568.102053:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698568.102057:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102064:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698568.102067:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102074:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698568.102083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698568.102085:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698568.102087:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126354944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698568.102095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698568.102106:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102110:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698568.102116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698568.102119:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102124:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698568.102127:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102133:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102138:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698568.102143:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698568.102144:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698568.102146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698568.102153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102158:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698568.102165:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102169:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698568.102173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698568.102177:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102187:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698568.102189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102196:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698568.102204:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698568.102206:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698568.102208:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698568.102215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698568.102226:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102230:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698568.102234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698568.102237:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698593 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102242:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698568.102245:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698568/real 1787698568] req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698568.102256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698568.102261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698568.102262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698568.102264:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698568.102268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698568.102271:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126354944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698568.102275:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698568.102278:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126355072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698568.102282:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698568.102285:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126355200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698568.102291:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698568.102296:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126355328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698593 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698568.102302:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698568.613495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698568.613495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698568.613497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698568.741592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698569.637545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698569.637545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698569.637545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698569.765556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698570.021519:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301143407, at 4301143424 00000100:00000040:3.0:1787698570.021519:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301143407, at 4301143424 00000100:00100000:3.0:1787698570.661464:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698570.661491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698570.661494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698570.789654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698571.685495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698571.685495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698571.685500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698571.813682:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698572.709537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698572.709537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698572.709537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698572.837623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698573.221540:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698573.221552:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698573.221556:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698573.221594:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6474 00000100:00080000:1.0:1787698573.221607:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698573.221611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221615:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221623:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698573.221628:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698573.221630:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221632:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221635:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698573.221637:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221652:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698573.221679:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698573.221692:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698573.221716:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698573.221719:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698573.221720:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698573.221723:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6474 00000100:00080000:1.0:1787698573.221726:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698573.221727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221731:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698573.221734:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698573.221737:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221739:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221741:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698573.221743:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698573.221751:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698573.221754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698573.221756:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698573.221768:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698573.221772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126355456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698573.221782:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698573.221789:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698573.221791:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698573.221792:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698573.221794:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6474 00000100:00080000:1.0:1787698573.221797:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698573.221799:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221800:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221802:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698573.221803:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698573.221807:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698573.221809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221814:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221818:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698573.221823:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698573.221827:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787698573.221829:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698573.221833:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698573.221841:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698573.221847:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698573.221851:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698573.221854:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698573.221855:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698573.221858:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6474 00000100:00080000:1.0:1787698573.221860:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698573.221862:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221864:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221867:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000800:00000100:2.0:1787698573.221868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221870:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698573.221874:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221877:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221878:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698573.221880:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698573.221884:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698573.221888:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698573.221889:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698573.221900:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698573.221906:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698573.221913:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698573.221916:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698573.221918:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698573.221921:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6474 00000100:00080000:1.0:1787698573.221925:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698573.221930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698573/real 1787698573] req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698573.221931:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698573.221939:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221945:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698573.221948:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698573.221953:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698573.221954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698573.221962:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698573.221966:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787698573.221969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698573.221971:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698573.221981:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698573.221982:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698573.221987:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787698573.221988:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698573.221991:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698573.222006:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698573.222006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126355456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698573.222019:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126355456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698573.222025:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698573.222044:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6484) 00000020:00000040:2.0:1787698573.222044:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698573.222063:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222071:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698573.222082:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222087:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698573.222095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698573.222100:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222106:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698573.222110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698573/real 1787698573] req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222117:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698573.222126:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698573.222128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698573.222131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698573.222139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698573.222150:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222154:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698573.222159:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698573.222162:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222168:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698573.222171:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698573/real 1787698573] req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222177:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698573.222187:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698573.222188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698573.222190:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698573.222202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698573.222213:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222217:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698573.222221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698573.222225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222229:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698573.222232:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698573/real 1787698573] req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222239:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698573.222248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698573.222249:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698573.222251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698573.222258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222264:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698573.222270:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222276:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698573.222284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698573.222290:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698598 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222298:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698573.222304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698573/real 1787698573] req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222315:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698573.222323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698573.222330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698573.222333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698573.222337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698573.222344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126355968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698573.222350:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126355584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698573.222364:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698573.222369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126355712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698573.222375:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698573.222379:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126355840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698573.222383:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698573.222386:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126355968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698598 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698573.222390:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698573.733505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698573.733505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698573.733512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698573.861613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698574.757553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698574.757562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698574.757577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698574.885643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698575.141574:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301148527, at 4301148544 00000100:00000040:3.0:1787698575.141574:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301148527, at 4301148544 00000100:00100000:0.0:1787698575.781589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698575.781589:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698575.781627:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698575.909591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698576.805566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698576.805568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698576.805577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698576.933638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698577.829573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698577.829573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698577.829574:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698577.957562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698578.341565:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698578.341575:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698578.341579:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698578.341589:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6479 00000100:00080000:3.0:1787698578.341604:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698578.341606:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.341610:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.341613:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698578.341618:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698578.341620:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.341622:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.341624:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698578.341645:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698578.341656:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698578.341681:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698578.341694:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698578.341714:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698578.341717:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698578.341719:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698578.341726:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6479 00000100:00080000:3.0:1787698578.341729:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698578.341732:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.341734:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.341736:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698578.341739:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698578.341741:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.341743:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.341745:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698578.341750:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698578.341751:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.341766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126356096:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698578.341771:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698578.341778:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698578.341791:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698578.341798:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698578.341809:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.341810:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698578.341834:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698578.341837:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698578.341839:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698578.341845:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6479 00000800:00000100:1.0:1787698578.341848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698578.341851:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698578.341862:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.341865:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698578.341868:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698578.341868:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698578.341872:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698578.341879:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.341882:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.341884:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698578.341886:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698578.341890:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698578.341894:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698578.341913:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698578/real 1787698578] req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698578.341913:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698578.341934:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698578.341940:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698578.341943:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698578.341944:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698578.341948:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698578.341950:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6479 00000100:00080000:3.0:1787698578.341955:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698578.341965:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698578.341966:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.341972:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.341976:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698578.341980:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:1.0:1787698578.341988:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698578.341994:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698578.341996:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698578.342002:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698578.342004:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698578.342006:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698578.342014:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698578.342023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126356096:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698578.342024:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698578.342033:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698578.342034:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126356096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698578.342046:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787698578.342048:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698578.342061:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342071:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698578.342071:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698578.342077:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698578.342079:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698578.342086:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6479 00000100:00080000:3.0:1787698578.342092:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698578.342094:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698578.342096:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.342102:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.342103:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698578.342105:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698578.342107:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698578.342108:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698578.342110:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698578.342116:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698578.342119:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698578.342129:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698578.342130:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698578.342143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698578.342143:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698578.342154:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698578.342159:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698578.342168:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787698578.342175:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698578.342184:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6489) 00000100:00100000:1.0:1787698578.342191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698578/real 1787698578] req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342213:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698578.342220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698578.342223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698578.342228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698578.342242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342250:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698578.342263:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342271:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698578.342282:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698578.342288:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342297:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698578.342302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698578/real 1787698578] req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342320:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698578.342325:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698578.342327:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698578.342330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342339:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698578.342342:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126356224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698578.342347:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698578.342352:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126356352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698578.342356:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698578.342361:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342366:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698578.342376:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342379:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698578.342386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698578.342390:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342396:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698578.342399:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698578/real 1787698578] req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342405:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698578.342415:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698578.342417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698578.342436:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698578.342448:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698578.342467:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342474:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698578.342495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698578.342501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698603 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342510:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698578.342515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698578/real 1787698578] req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698578.342534:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698578.342541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698578.342545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698578.342549:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698578.342556:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698578.342561:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126356480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698578.342566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698578.342569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126356608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698603 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698578.342581:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698578.853537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698578.853537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698578.853537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698578.981604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698579.877571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698579.877572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698579.877577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698580.005621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698580.261576:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301153647, at 4301153664 00000100:00000040:3.0:1787698580.261576:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301153647, at 4301153664 00000100:00100000:0.0:1787698580.901569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698580.901569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698580.901570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698581.029594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698581.925571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698581.925572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698581.925573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698582.053541:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698582.949556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698582.949556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698582.949556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698583.077577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698583.461538:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698583.461547:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698583.461551:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698583.461560:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6484 00000100:00080000:2.0:1787698583.461573:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698583.461575:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461578:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461581:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698583.461586:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698583.461588:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461590:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461592:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698583.461594:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461609:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698583.461634:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698583.461647:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698583.461665:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698583.461667:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698583.461668:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698583.461671:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6484 00000100:00080000:2.0:1787698583.461673:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698583.461675:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461676:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461678:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698583.461679:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698583.461681:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461683:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461684:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698583.461686:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698583.461695:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698583.461698:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698583.461704:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698583.461711:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.461718:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698583.461719:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698583.461724:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698583.461727:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698583.461733:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6484 00000100:00080000:2.0:1787698583.461738:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698583.461739:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461740:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461742:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698583.461744:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00100000:1.0:1787698583.461745:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126356736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698583.461748:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461754:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461755:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698583.461757:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698583.461759:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698583.461763:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698583.461770:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698583.461775:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.461785:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698583.461787:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698583.461791:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698583.461793:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698583.461800:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6484 00000100:00080000:2.0:1787698583.461803:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698583.461807:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461807:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461811:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461812:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698583.461813:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698583.461815:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461816:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461818:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698583.461819:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698583.461823:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698583.461826:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698583.461839:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698583.461845:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698583.461849:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698583.461851:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698583.461853:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698583.461855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698583.461858:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6484 00000100:00080000:2.0:1787698583.461862:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698583.461864:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461866:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461868:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698583.461869:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698583.461872:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698583.461874:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698583.461875:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698583.461879:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698583.461880:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698583.461882:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698583.461887:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698583.461894:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698583.461912:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698583.461921:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6494) 00000100:00100000:1.0:1787698583.461940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698583/real 1787698583] req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.461956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.461965:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698583.461973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698583.461977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698583.461980:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.461988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126356736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698583.461994:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126356736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698583.462004:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698583.462019:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462027:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698583.462038:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462045:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698583.462057:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698583.462063:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462078:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698583.462084:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698583/real 1787698583] req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698583.462111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698583.462113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698583.462117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462123:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698583.462128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698583.462146:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462153:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698583.462162:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698583.462169:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462178:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698583.462184:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698583/real 1787698583] req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462194:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698583.462208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698583.462211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698583.462215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462222:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126356992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698583.462227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698583.462246:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462253:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698583.462261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698583.462267:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462275:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698583.462281:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698583/real 1787698583] req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698583.462314:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698583.462317:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698583.462320:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462328:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698583.462333:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698583.462350:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462357:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698583.462367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698583.462373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698608 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462382:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698583.462387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698583/real 1787698583] req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462449:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698583.462458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698583.462464:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698583.462467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698583.462471:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698583.462478:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698583.462483:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126356864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698583.462491:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698583.462497:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126356992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698583.462506:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698583.462510:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126357120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698583.462517:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698583.462521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126357248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698608 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698583.462527:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698583.973524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698583.973530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698583.973662:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698584.101604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698584.997543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698584.997543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698584.997557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698585.125645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698585.381546:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301158767, at 4301158784 00000100:00000040:3.0:1787698585.381546:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301158767, at 4301158784 00000100:00100000:0.0:1787698586.021486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698586.021486:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698586.021486:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698586.149655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698587.045549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698587.045549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698587.045549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698587.173590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698588.069554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698588.069554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698588.069655:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698588.197614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698588.581551:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698588.581561:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698588.581565:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698588.581575:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6489 00000100:00080000:3.0:1787698588.581586:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698588.581589:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581592:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581595:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698588.581600:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698588.581602:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581604:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581606:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698588.581628:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581641:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698588.581666:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698588.581679:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698588.581701:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698588.581706:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698588.581708:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698588.581712:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6489 00000100:00080000:3.0:1787698588.581718:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698588.581721:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581725:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581728:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698588.581731:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698588.581736:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698588.581738:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698588.581740:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581742:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698588.581743:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698588.581754:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698588.581759:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698588.581760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126357376:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698588.581770:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698588.581777:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698588.581791:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698588.581795:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698588.581801:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:1.0:1787698588.581804:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698588.581805:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698588.581811:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6489 00000100:00080000:3.0:1787698588.581814:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698588.581816:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581818:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581819:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698588.581821:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698588.581823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581825:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581826:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698588.581828:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698588.581831:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698588.581834:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698588.581841:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698588.581847:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698588.581852:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698588.581854:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698588.581855:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698588.581858:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6489 00000100:00080000:3.0:1787698588.581860:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698588.581865:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581867:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581869:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698588.581870:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698588.581872:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581873:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581874:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698588.581876:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698588.581880:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698588.581883:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698588.581891:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698588.581897:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698588.581902:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698588.581904:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698588.581905:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698588.581908:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6489 00000100:00000040:1.0:1787698588.581911:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698588.581912:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698588.581917:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581921:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581926:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698588.581929:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698588.581933:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698588.581937:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698588.581937:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698588.581942:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698588.581949:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698588.581952:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698588.581956:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698588.581960:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698588.581964:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698588.581989:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698588.581989:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698588.582000:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6499) 00000100:00000040:1.0:1787698588.582010:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787698588.582020:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698588.582030:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698588.582043:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698588.582051:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698588.582065:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787698588.582072:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698588.582081:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698588.582093:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698588.582100:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698588.582111:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787698588.582120:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698588.582129:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698588.582140:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698588.582147:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698588.582158:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698589.093549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698589.093549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698589.093562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698589.221568:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698590.117569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698590.117569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698590.117582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698590.245615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698590.501522:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301163887, at 4301163904 00000100:00000040:2.0:1787698590.501534:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301163887, at 4301163904 00000100:00100000:2.0:1787698591.141525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698591.141525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698591.141537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698591.269566:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698591.653874:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698591.653940:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698591.653960:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698591.654012:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698591.654035:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698591.654044:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698591.654054:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698591.654063:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698591.654072:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698591.654081:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698591.654117:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698591.654127:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698613 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698588/real 1787698591] req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698591.654529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698591.654540:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698591.654547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698591.654551:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357888:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698591.654573:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698588/real 1787698591] req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654585:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698591.654594:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698591.654602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698591.654605:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698591.654609:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357504:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698591.654628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698588/real 1787698591] req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654641:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698591.654649:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698591.654662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698591.654666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698591.654670:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654678:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357632:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698591.654684:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698588/real 1787698591] req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654694:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698591.654702:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698591.654709:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698591.654712:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698591.654716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654723:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126357760:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698591.654728:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698588/real 1787698591] req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654739:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698591.654757:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698591.654764:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698591.654768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698591.654771:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698591.654778:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126357376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698591.654784:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126357888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698591.654798:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698591.654808:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126357504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698591.654816:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698591.654821:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126357632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698591.654828:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698591.654833:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126357760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698591.654840:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698591.654845:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126357376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698613 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698591.654852:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698592.165534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698592.165534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698592.165534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698592.293635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698593.189529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698593.189529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698593.189540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698593.317599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698593.701543:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698593.701553:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698593.701556:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698593.701566:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6494 00000100:00080000:3.0:1787698593.701577:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698593.701580:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701583:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701586:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698593.701591:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698593.701594:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701595:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701598:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698593.701600:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701611:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698593.701635:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698593.701648:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698593.701667:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698593.701670:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698593.701687:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698593.701692:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6494 00000100:00080000:3.0:1787698593.701696:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698593.701700:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701703:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701706:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698593.701709:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698593.701713:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698593.701719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698593.701720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701725:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698593.701730:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698593.701744:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698593.701747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126358016:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698593.701752:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698593.701766:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698593.701777:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698593.701786:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698593.701790:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698593.701791:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698593.701796:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6494 00000100:00000040:1.0:1787698593.701801:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698593.701802:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698593.701809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698593.701820:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701821:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701826:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698593.701829:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698593.701840:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701844:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701846:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698593.701849:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698593.701854:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698593.701860:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698593.701871:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698593.701884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701885:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698593.701897:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698593.701900:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698593.701903:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698593.701911:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698593.701915:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6494 00000100:00080000:3.0:1787698593.701922:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698593.701928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701931:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701934:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698593.701937:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698593.701940:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698593.701943:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.701946:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698593.701948:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698593.701954:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698593.701960:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698593.701965:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698593/real 1787698593] req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698593.701974:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698593.701989:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698593.701997:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702006:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698593.702010:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698593.702020:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698593.702024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698593.702031:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:1.0:1787698593.702033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698593.702035:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698593.702043:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6494 00000100:00100000:1.0:1787698593.702048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126358016:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698593.702051:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698593.702057:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698593.702060:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126358016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698593.702064:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.702071:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698593.702076:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698593.702080:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698593.702081:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698593.702083:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698593.702088:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698593.702090:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698593.702098:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698593.702104:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698593.702107:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698593.702117:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698593.702121:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698593.702138:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702145:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702156:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698593.702158:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6504) 00000800:00000100:1.0:1787698593.702173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698593.702182:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702195:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698593.702201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698593/real 1787698593] req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698593.702240:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698593.702243:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698593.702247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702253:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698593.702258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698593.702284:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702291:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698593.702301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698593.702307:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702315:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698593.702319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698593/real 1787698593] req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698593.702341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698593.702344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698593.702347:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698593.702358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698593.702377:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702383:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698593.702391:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698593.702396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702404:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698593.702408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698593/real 1787698593] req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698593.702461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698593.702464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698593.702468:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698593.702495:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126358144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698593.702505:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698593.702512:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126358272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698593.702526:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698593.702532:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126358400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698593.702540:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698593.702548:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698593.702570:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702577:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698593.702588:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698593.702597:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698618 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702606:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698593.702612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698593/real 1787698593] req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702624:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698593.702632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698593.702639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698593.702642:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698593.702646:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698593.702653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698593.702659:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126358528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698618 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698593.702666:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698594.213530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698594.213530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698594.213530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698594.341647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698595.237558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698595.237560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698595.237560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698595.365578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698595.621505:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301169007, at 4301169024 00000100:00000040:3.0:1787698595.621515:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301169007, at 4301169024 00000100:00100000:2.0:1787698596.261496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698596.261552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698596.261552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698596.389626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698597.285568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698597.285568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698597.285568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698597.413599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698598.309496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698598.309497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698598.309505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698598.437635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698598.821539:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698598.821551:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698598.821555:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698598.821566:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6499 00000100:00080000:3.0:1787698598.821579:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698598.821582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821586:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821590:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698598.821596:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698598.821600:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821604:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821607:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698598.821610:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821628:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698598.821663:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698598.821678:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698598.821700:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698598.821702:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698598.821704:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698598.821707:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6499 00000100:00080000:3.0:1787698598.821710:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698598.821712:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821715:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698598.821734:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698598.821737:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821739:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821741:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698598.821744:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698598.821753:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698598.821755:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698598.821758:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698598.821772:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698598.821779:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698598.821787:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698598.821788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126358656:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698598.821799:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698598.821801:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698598.821805:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6499 00000100:00080000:3.0:1787698598.821808:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698598.821810:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821813:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821814:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698598.821816:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698598.821819:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821820:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821822:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698598.821827:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698598.821829:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698598.821831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698598.821839:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698598.821849:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787698598.821857:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698598.821865:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698598.821871:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698598.821873:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698598.821874:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698598.821877:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6499 00000100:00080000:3.0:1787698598.821887:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698598.821889:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821893:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698598.821897:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:1.0:1787698598.821898:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821902:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821906:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821908:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698598.821910:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698598.821914:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698598.821918:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698598.821926:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698598.821927:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698598.821963:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698598.821969:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698598.821971:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698598.821976:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698598.821979:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6499 00000100:00080000:3.0:1787698598.821983:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698598.821986:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.821990:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.821992:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698598.821996:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698598.822001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698598/real 1787698598] req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698598.822002:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698598.822010:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698598.822017:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698598.822025:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698598.822032:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698598.822038:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698598.822045:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698598.822052:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698598.822083:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822091:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698598.822096:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6509) 00000100:00080000:1.0:1787698598.822108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698598.822114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698598.822120:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822145:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126358656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698598.822154:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126358656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698598.822167:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698598.822188:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698598.822216:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822225:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698598.822240:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698598.822249:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822261:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698598.822268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698598/real 1787698598] req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822288:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698598.822307:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698598.822311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698598.822315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698598.822330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698598.822353:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822361:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698598.822375:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698598.822382:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822392:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698598.822398:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698598/real 1787698598] req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822416:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698598.822468:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698598.822472:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698598.822477:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822484:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126358912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698598.822488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822494:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698598.822506:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822511:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698598.822521:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698598.822527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822534:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698598.822538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698598/real 1787698598] req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822566:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698598.822572:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698598.822573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698598.822576:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698598.822584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698598.822597:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822603:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698598.822609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698598.822614:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698623 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822620:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698598.822623:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698598/real 1787698598] req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822636:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698598.822643:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698598.822648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698598.822650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698598.822652:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698598.822657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698598.822661:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126358784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698598.822666:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698598.822671:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126358912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698598.822675:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698598.822678:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126359040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698598.822683:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698598.822687:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126359168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698623 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698598.822691:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698599.333524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698599.333526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698599.333604:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698599.461596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698600.357516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698600.357516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698600.357555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698600.485666:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698600.741545:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301174127, at 4301174144 00000100:00000040:3.0:1787698600.741545:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301174127, at 4301174144 00000100:00100000:1.0:1787698601.381556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698601.381556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698601.381557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698601.509567:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698602.405540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698602.405540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698602.405540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698602.533613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698603.429537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698603.429537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698603.429588:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698603.557598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698603.941582:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698603.941592:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698603.941597:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698603.941607:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6504 00000100:00080000:0.0:1787698603.941618:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698603.941622:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941628:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698603.941633:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698603.941636:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941641:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698603.941643:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941658:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698603.941684:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698603.941697:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698603.941717:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698603.941720:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698603.941722:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698603.941725:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6504 00000100:00080000:0.0:1787698603.941728:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698603.941729:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941731:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941733:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698603.941735:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698603.941737:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941739:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941740:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698603.941742:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698603.941751:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698603.941755:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698603.941763:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698603.941774:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.941776:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698603.941781:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698603.941786:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698603.941789:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698603.941796:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6504 00000100:00080000:0.0:1787698603.941803:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698603.941805:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941811:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941813:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00100000:1.0:1787698603.941813:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126359296:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698603.941816:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698603.941822:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941825:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941827:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698603.941829:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698603.941832:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698603.941835:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698603.941841:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698603.941848:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698603.941854:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698603.941860:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:1.0:1787698603.941860:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698603.941863:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698603.941871:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6504 00000100:00080000:0.0:1787698603.941875:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698603.941879:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941882:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941887:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698603.941889:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698603.941891:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941894:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698603.941896:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698603.941899:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698603.941903:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698603.941910:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698603.941916:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698603.941922:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698603.941926:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698603.941927:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698603.941931:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698603.941935:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6504 00000100:00080000:0.0:1787698603.941938:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698603.941940:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941942:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698603.941944:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698603.941945:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698603.941948:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698603.941953:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698603.941954:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698603.941955:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698603.941960:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698603.941965:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698603.941971:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698603.941979:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698603.941985:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698603.941993:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6515) 00000100:00100000:1.0:1787698603.942009:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698603/real 1787698603] req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942052:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698603.942063:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698603.942067:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698603.942071:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942080:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126359296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698603.942087:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126359296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698603.942104:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698603.942122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698603.942147:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942163:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698603.942178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698603.942186:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942197:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698603.942203:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698603/real 1787698603] req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698603.942234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698603.942238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698603.942242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698603.942260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942270:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698603.942283:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942289:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698603.942298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698603.942303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942310:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698603.942314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698603/real 1787698603] req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942322:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698603.942334:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698603.942335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698603.942338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942342:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698603.942346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942353:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698603.942361:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942366:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698603.942372:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698603.942376:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942382:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698603.942385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698603/real 1787698603] req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698603.942408:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698603.942411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698603.942414:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698603.942460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698603.942481:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942486:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698603.942494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698603.942499:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698628 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942505:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698603.942509:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698603/real 1787698603] req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698603.942523:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698603.942528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698603.942530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698603.942532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698603.942536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126359808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698603.942540:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126359424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698603.942549:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698603.942556:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126359552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698603.942571:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698603.942577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126359680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698603.942585:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698603.942591:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126359808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698628 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698603.942599:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698604.453559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698604.453559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698604.453559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698604.581864:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698605.477515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698605.477553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698605.477562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698605.605644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698605.861487:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301179247, at 4301179264 00000100:00000040:3.0:1787698605.861525:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301179247, at 4301179264 00000100:00100000:2.0:1787698606.501543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698606.501545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698606.501555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698606.629569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698607.525539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698607.525539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698607.525594:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698607.653612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698608.549541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698608.549542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698608.549548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698608.677573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698609.061520:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698609.061533:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698609.061538:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698609.061556:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6510 00000100:00080000:1.0:1787698609.061570:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698609.061574:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.061578:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061582:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698609.061588:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698609.061592:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.061594:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061597:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698609.061600:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061617:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698609.061646:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698609.061663:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.061689:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698609.061693:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698609.061695:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698609.061699:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6510 00000100:00080000:1.0:1787698609.061703:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698609.061707:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.061711:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061713:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698609.061716:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698609.061721:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698609.061722:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.061728:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061734:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698609.061740:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698609.061750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126359936:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787698609.061752:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698609.061760:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698609.061772:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698609.061783:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.061789:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.061799:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698609.061804:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698609.061808:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698609.061811:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698609.061816:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6510 00000100:00080000:1.0:1787698609.061820:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698609.061823:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.061826:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061829:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698609.061840:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698609.061844:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.061846:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:2.0:1787698609.061850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061851:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698609.061857:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698609.061863:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698609.061868:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698609.061870:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.061881:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698609.061892:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.061901:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698609.061905:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698609.061907:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698609.061914:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6510 00000100:00080000:1.0:1787698609.061918:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698609.061919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698609.061923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.061930:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061935:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698609.061938:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698609.061941:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698609.061944:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698609.061948:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.061967:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698609.061969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.061973:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787698609.061979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787698609.061982:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698609.061984:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698609.061990:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698609.061992:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698609.061999:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126359936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698609.062005:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126359936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698609.062011:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787698609.062015:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698609.062026:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062032:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062049:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360064:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698609.062050:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698609.062056:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698609.062059:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698609.062065:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6510 00000100:00000040:2.0:1787698609.062066:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698609.062072:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698609.062076:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698609.062080:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:2.0:1787698609.062087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698609.062093:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698609.062098:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.062102:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698609.062103:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000020:00000040:1.0:1787698609.062105:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:2.0:1787698609.062110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698609.062113:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698609.062120:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698609.062125:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698609.062132:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698609.062138:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698609.062139:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698609.062148:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698609.062150:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698609.062161:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698609.062165:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698609.062168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698609.062177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698609.062187:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698609.062205:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698609.062209:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6520) 00000100:00100000:2.0:1787698609.062222:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698609.062234:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062242:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698609.062250:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698609.062255:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062261:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698609.062264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062280:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698609.062285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698609.062288:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698609.062291:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698609.062303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126360064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698609.062311:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698609.062317:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126360192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698609.062324:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698609.062332:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698609.062349:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062356:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698609.062367:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698609.062375:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062383:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698609.062390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698609.062438:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698609.062442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698609.062446:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698609.062459:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062468:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698609.062480:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062487:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698609.062504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698609.062511:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698634 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062525:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698609.062531:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698609/real 1787698609] req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062542:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698609.062550:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698609.062557:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698609.062560:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698609.062563:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698609.062571:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698609.062576:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126360320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698609.062584:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698609.062590:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126360448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698634 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698609.062596:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698609.573521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698609.573522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698609.573522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698609.701620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698610.597546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698610.597546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698610.597828:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698610.725614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698610.981521:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301184367, at 4301184384 00000100:00000040:3.0:1787698610.981521:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301184367, at 4301184384 00000100:00100000:0.0:1787698611.621542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698611.621542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698611.621542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698611.749577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698612.645567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698612.645567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698612.645749:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698612.773660:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698613.669518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698613.669518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698613.669551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698613.797636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698614.181517:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698614.181526:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698614.181530:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698614.181540:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6515 00000100:00080000:1.0:1787698614.181551:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698614.181555:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181558:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181562:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698614.181567:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698614.181570:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181573:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181575:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698614.181579:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181593:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698614.181617:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698614.181638:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698614.181668:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698614.181672:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698614.181673:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698614.181678:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6515 00000100:00080000:1.0:1787698614.181682:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698614.181684:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181687:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181689:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698614.181692:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698614.181695:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181697:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181699:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698614.181702:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698614.181711:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698614.181717:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698614.181740:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698614.181750:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698614.181758:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698614.181761:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698614.181762:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698614.181766:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6515 00000100:00080000:1.0:1787698614.181770:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698614.181772:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181775:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181777:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698614.181779:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698614.181788:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181800:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181803:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698614.181805:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698614.181810:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698614.181815:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698614.181824:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698614.181833:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698614.181841:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698614.181844:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698614.181846:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698614.181850:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6515 00000100:00080000:1.0:1787698614.181854:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698614.181856:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181859:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181861:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698614.181871:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698614.181875:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181877:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181879:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698614.181882:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698614.181888:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698614.181892:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698614.181900:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698614.181910:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698614.181917:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698614.181921:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698614.181922:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698614.181925:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6515 00000100:00080000:1.0:1787698614.181929:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698614.181931:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181934:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181936:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698614.181938:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698614.181941:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698614.181943:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698614.181945:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698614.181947:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698614.181951:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698614.181955:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698614.181962:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698614.181970:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698614.181979:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6525) 00000100:00000040:2.0:1787698614.182061:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182075:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126360576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698614.182102:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182110:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698614.182143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698614.182158:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182189:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698614/real 1787698614] req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698614.182220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698614.182224:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698614.182228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126360576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698614.182241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698614.182262:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182269:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698614.182281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698614.182288:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182298:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698614.182304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698614/real 1787698614] req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698614.182343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698614.182347:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698614.182350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698614.182363:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698614.182384:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182392:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698614.182405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698614.182412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182442:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698614.182449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698614/real 1787698614] req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182461:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182469:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698614.182477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698614.182479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698614.182483:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698614.182496:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182504:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698614.182517:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182523:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698614.182535:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698614.182542:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182557:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698614.182562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698614/real 1787698614] req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182582:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698614.182589:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698614.182592:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698614.182596:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182603:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126360960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698614.182609:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182617:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698614.182629:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182636:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698614.182648:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698614.182655:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698639 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182663:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698614.182669:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698614/real 1787698614] req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182680:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698614.182688:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698614.182695:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698614.182698:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698614.182701:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698614.182708:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698614.182714:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126360576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698614.182723:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698614.182731:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126360704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698614.182738:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698614.182743:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126360832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698614.182750:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698614.182754:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126360960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698614.182761:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698614.182766:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126361088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698639 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698614.182772:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698614.693532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698614.693532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698614.693537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698614.821575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698615.717543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698615.717548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698615.717551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698615.845626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698616.101546:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301189487, at 4301189504 00000100:00000040:3.0:1787698616.101549:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301189487, at 4301189504 00000100:00100000:3.0:1787698616.741551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698616.741561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698616.741574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698616.869920:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698617.765538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698617.765551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698617.765551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698617.893639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698618.789543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698618.789543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698618.789562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698618.917593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698619.301571:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698619.301581:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698619.301585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698619.301598:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6520 00000100:00080000:0.0:1787698619.301611:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698619.301615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301620:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301625:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698619.301631:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698619.301634:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301637:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301640:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698619.301644:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301661:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698619.301691:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698619.301703:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.301728:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698619.301730:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698619.301732:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698619.301735:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6520 00000100:00080000:0.0:1787698619.301738:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698619.301739:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301741:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301742:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698619.301745:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698619.301748:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301750:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301752:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698619.301754:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698619.301763:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698619.301768:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698619.301770:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698619.301792:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698619.301798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126361216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698619.301812:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.301823:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698619.301826:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698619.301827:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698619.301830:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6520 00000100:00000040:2.0:1787698619.301834:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.301836:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698619.301842:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698619.301849:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301854:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301858:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698619.301860:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698619.301862:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301864:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301866:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698619.301868:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698619.301872:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698619.301875:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698619.301882:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698619.301890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301894:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.301900:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698619.301902:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698619.301903:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698619.301907:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6520 00000100:00000040:2.0:1787698619.301910:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.301911:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698619.301917:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301921:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301923:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698619.301924:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698619.301928:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.301929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.301931:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698619.301933:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698619.301936:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698619.301941:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698619.301946:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698619.301955:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.301955:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698619/real 1787698619] req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.301982:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698619.301985:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698619.301986:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698619.301990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.301991:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6520 00000100:00080000:0.0:1787698619.301995:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698619.302000:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698619.302005:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698619.302007:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698619.302010:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698619.302010:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698619.302019:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698619.302021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698619.302025:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698619.302028:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787698619.302032:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787698619.302033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698619.302039:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698619.302044:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126361216:192.168.201.136@tcp:250:kworker.0 00000008:00100000:0.0:1787698619.302048:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698619.302050:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126361216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698619.302056:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:2.0:1787698619.302059:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698619.302072:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698619.302076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698619.302088:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302091:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698619.302107:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6530) 00000100:00000040:2.0:1787698619.302107:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302114:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698619.302124:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698619.302128:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302136:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698619.302139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698619/real 1787698619] req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698619.302162:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698619.302164:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698619.302166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698619.302173:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302178:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698619.302184:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302189:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698619.302194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698619.302197:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302203:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698619.302206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698619/real 1787698619] req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302225:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698619.302229:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698619.302232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698619.302234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698619.302242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302246:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698619.302253:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302258:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698619.302263:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698619.302266:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302272:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698619.302275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698619/real 1787698619] req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302281:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698619.302290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698619.302292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698619.302293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302299:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698619.302302:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126361344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698619.302306:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698619.302310:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126361472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698619.302313:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698619.302316:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126361600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698619.302321:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698619.302325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698619.302335:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302340:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698619.302345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698619.302348:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698644 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302354:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698619.302357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698619/real 1787698619] req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302367:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698619.302372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698619.302375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698619.302377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698619.302379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698619.302384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698619.302387:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126361728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698644 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698619.302390:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698619.813549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698619.813556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698619.813583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698619.941652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698620.837550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698620.837551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698620.837655:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698620.965584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698621.221554:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301194607, at 4301194624 00000100:00000040:3.0:1787698621.221554:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301194607, at 4301194624 00000100:00100000:0.0:1787698621.861544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698621.861544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698621.861544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698621.989583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698622.885514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698622.885515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698622.885515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698623.013614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698623.909497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698623.909497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698623.909502:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698624.037551:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698624.421538:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698624.421554:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698624.421560:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698624.421579:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6525 00000100:00080000:0.0:1787698624.421592:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698624.421597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421605:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698624.421611:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698624.421615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421617:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421620:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698624.421623:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421641:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698624.421672:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698624.421698:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698624.421723:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698624.421726:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698624.421729:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698624.421733:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6525 00000100:00080000:0.0:1787698624.421736:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698624.421739:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421741:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421744:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698624.421747:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698624.421750:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421753:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421756:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698624.421759:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698624.421768:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698624.421770:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698624.421779:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698624.421787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126361856:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698624.421792:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698624.421803:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698624.421812:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698624.421816:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787698624.421818:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698624.421819:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698624.421828:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6525 00000100:00080000:0.0:1787698624.421836:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698624.421840:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421844:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421848:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421851:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698624.421854:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698624.421857:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421860:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421862:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698624.421865:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698624.421871:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698624.421876:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787698624.421879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698624.421890:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698624.421898:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698624.421909:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698624.421919:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698624.421924:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698624.421926:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698624.421931:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6525 00000100:00080000:0.0:1787698624.421935:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698624.421938:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421944:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698624.421945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698624/real 1787698624] req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698624.421947:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698624.421951:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698624.421958:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.421965:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.421968:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698624.421970:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698624.421976:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698624.421981:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698624.421985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698624.421991:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698624.422000:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698624.422011:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698624.422011:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698624.422019:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698624.422026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698624.422029:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698624.422037:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698624.422041:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698624.422042:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126361856:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698624.422049:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6525 00000100:00080000:0.0:1787698624.422054:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698624.422056:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126361856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698624.422060:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.422067:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.422070:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698624.422073:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698624.422075:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698624.422080:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698624.422083:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698624.422086:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698624.422086:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:0.0:1787698624.422094:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698624.422098:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698624.422108:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698624.422114:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698624.422121:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698624.422139:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6535) 00000100:00000040:2.0:1787698624.422153:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422163:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698624.422172:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698624.422177:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422183:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698624.422187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698624/real 1787698624] req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698624.422211:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698624.422212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698624.422215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126361984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698624.422223:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698624.422236:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422243:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698624.422253:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698624.422260:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422269:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698624.422274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698624/real 1787698624] req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698624.422295:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698624.422296:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698624.422298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422303:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698624.422306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422311:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698624.422319:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422323:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698624.422330:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698624.422334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698624.422346:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698624/real 1787698624] req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698624.422362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698624.422364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698624.422366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698624.422373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126361984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698624.422378:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698624.422381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126362112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698624.422385:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698624.422388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126362240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698624.422392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698624.422396:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698624.422408:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422412:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698624.422417:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698624.422450:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698649 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422457:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698624.422463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698624/real 1787698624] req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698624.422478:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698624.422491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698624.422493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698624.422497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698624.422504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698624.422509:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126362368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698649 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698624.422515:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698624.933478:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698624.933510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698624.933521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698625.061651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698625.957499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698625.957499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698625.957507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698626.085586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698626.341506:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301199727, at 4301199744 00000100:00000040:3.0:1787698626.341506:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301199727, at 4301199744 00000100:00100000:0.0:1787698626.981538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698626.981538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698626.981538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698627.109629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698628.005491:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698628.005495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698628.005606:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698628.133543:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698629.029482:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698629.029482:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698629.029489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698629.157580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698629.541533:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698629.541543:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698629.541548:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698629.541559:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6530 00000100:00080000:0.0:1787698629.541571:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698629.541573:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541577:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541580:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698629.541585:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698629.541588:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541591:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541593:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698629.541596:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541648:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698629.541682:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698629.541695:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698629.541716:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698629.541719:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698629.541720:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698629.541724:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6530 00000100:00080000:0.0:1787698629.541727:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698629.541729:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541731:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541733:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698629.541736:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698629.541738:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541741:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541743:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698629.541745:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698629.541755:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698629.541758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698629.541759:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698629.541771:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698629.541781:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.541786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126362496:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698629.541793:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698629.541796:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698629.541797:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698629.541800:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6530 00000100:00080000:0.0:1787698629.541802:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698629.541804:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541806:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541808:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698629.541809:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698629.541811:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541813:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541814:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698629.541816:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698629.541819:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698629.541822:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698629.541832:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698629.541833:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698629.541848:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.541863:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698629.541868:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698629.541872:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698629.541874:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698629.541878:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6530 00000100:00080000:0.0:1787698629.541881:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698629.541884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541887:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698629.541889:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698629.541891:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541893:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541894:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698629.541896:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698629.541899:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698629.541903:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698629.541909:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698629.541919:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698629.541919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698629.541928:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698629.541930:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698629.541932:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698629.541935:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6530 00000100:00080000:0.0:1787698629.541937:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698629.541939:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541941:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541943:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698629.541947:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698629.541948:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698629.541951:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698629.541957:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698629.541961:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698629.541967:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698629.541972:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698629.541976:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698629.541982:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698629.541988:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698629.541996:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6540) 00000100:00100000:2.0:1787698629.542006:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698629/real 1787698629] req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542039:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698629.542049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698629.542053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698629.542058:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126362496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698629.542074:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126362496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698629.542099:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698629.542124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542134:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698629.542150:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542158:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698629.542170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698629.542178:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542188:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698629.542195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698629/real 1787698629] req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542206:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698629.542222:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698629.542225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698629.542229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698629.542243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698629.542269:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542276:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698629.542300:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698629.542313:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542327:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698629.542333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698629/real 1787698629] req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542354:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698629.542362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698629.542366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698629.542370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542377:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698629.542383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542392:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698629.542406:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542413:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698629.542445:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698629.542452:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542461:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698629.542467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698629/real 1787698629] req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698629.542495:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698629.542498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698629.542503:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126362880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698629.542517:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542524:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698629.542543:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542557:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698629.542569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698629.542576:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698654 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542584:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698629.542590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698629/real 1787698629] req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542625:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698629.542633:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698629.542641:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698629.542644:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698629.542648:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698629.542656:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698629.542662:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126362624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698629.542670:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698629.542677:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126362752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698629.542684:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698629.542690:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126362880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698629.542698:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698629.542704:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126363008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698654 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698629.542711:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698630.053526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698630.053526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698630.053527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698630.181558:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698631.077527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698631.077528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698631.077671:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698631.205643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698631.461558:0:88446:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301204847, at 4301204864 00000100:00000040:3.0:1787698631.461558:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301204847, at 4301204864 00000100:00100000:0.0:1787698632.101571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698632.101571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698632.101584:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698632.229645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698633.125538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698633.125538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698633.125538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698633.253675:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698634.149581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698634.149586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698634.149587:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698634.277656:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698634.661614:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698634.661624:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698634.661628:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698634.661640:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6535 00000100:00080000:0.0:1787698634.661653:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698634.661657:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.661662:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661666:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698634.661672:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698634.661676:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.661679:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661682:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698634.661686:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661705:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698634.661735:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698634.661754:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698634.661780:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698634.661784:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698634.661786:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698634.661791:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6535 00000100:00080000:0.0:1787698634.661795:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698634.661799:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.661802:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661804:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698634.661809:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698634.661812:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.661815:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661819:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698634.661823:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698634.661846:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698634.661855:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698634.661857:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698634.661871:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698634.661876:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126363136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698634.661899:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698634.661911:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698634.661917:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787698634.661917:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698634.661925:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698634.661932:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698634.661937:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6535 00000100:00080000:0.0:1787698634.661943:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698634.661948:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.661952:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661963:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000800:00000100:2.0:1787698634.661969:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661970:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698634.661978:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.661982:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.661986:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698634.661992:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698634.661994:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698634.662005:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698634.662015:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698634.662027:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698634.662031:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698634/real 1787698634] req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698634.662073:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662082:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698634.662092:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698634.662101:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698634.662103:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698634.662103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698634.662117:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6535 00000100:00080000:0.0:1787698634.662126:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698634.662127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698634.662135:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698634.662139:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698634.662142:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698634.662156:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.662166:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:2.0:1787698634.662170:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126363136:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698634.662171:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698634.662182:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.662187:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.662189:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698634.662194:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126363136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698634.662196:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698634.662211:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698634.662214:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:0.0:1787698634.662219:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698634.662229:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698634.662233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698634.662266:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363264:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698634.662297:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698634.662300:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698634.662303:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698634.662304:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698634.662315:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6535 00000100:00080000:0.0:1787698634.662325:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698634.662336:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698634.662336:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698634.662347:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.662349:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698634.662356:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:2.0:1787698634.662357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698634.662365:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698634.662371:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698634.662373:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698634.662375:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698634.662376:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698634.662380:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698634.662389:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698634.662394:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787698634.662400:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698634.662406:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698634/real 1787698634] req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698634.662471:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698634.662538:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6545) 00000100:00080000:2.0:1787698634.662541:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698634.662546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698634.662550:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662555:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698634.662560:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698634.662581:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662588:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698634.662598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698634.662604:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662611:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698634.662615:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698634/real 1787698634] req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662637:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698634.662645:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698634.662649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698634.662653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662666:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698634.662671:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126363264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698634.662676:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698634.662680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126363392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698634.662684:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698634.662690:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698634.662706:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662710:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698634.662718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698634.662722:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662728:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698634.662731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698634/real 1787698634] req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662737:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662742:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698634.662747:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698634.662749:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698634.662751:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698634.662758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698634.662768:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662772:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698634.662777:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698634.662780:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698659 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662786:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698634.662788:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698634/real 1787698634] req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662794:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698634.662800:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698634.662803:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698634.662805:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698634.662807:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698634.662812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698634.662815:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126363520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698634.662818:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698634.662821:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126363648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698659 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698634.662825:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698635.173556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698635.173556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698635.173556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698635.301653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698636.197562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698636.197562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698636.197568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698636.325606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698636.581576:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301209967, at 4301209984 00000100:00000040:3.0:1787698636.581581:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301209967, at 4301209984 00000100:00100000:3.0:1787698637.221513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698637.221514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698637.221538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698637.349690:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698638.245547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698638.245553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698638.245553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698638.373551:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698639.269536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698639.269536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698639.269536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698639.397576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698639.781549:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698639.781558:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698639.781562:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698639.781571:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6540 00000100:00080000:1.0:1787698639.781580:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698639.781583:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781586:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781589:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698639.781593:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698639.781596:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781597:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781600:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698639.781602:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781614:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698639.781638:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698639.781655:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698639.781673:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698639.781676:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698639.781677:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698639.781681:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6540 00000100:00080000:1.0:1787698639.781684:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698639.781686:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781688:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781690:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698639.781693:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698639.781695:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781700:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698639.781701:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698639.781702:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698639.781707:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698639.781718:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787698639.781722:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126363776:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698639.781724:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698639.781736:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698639.781745:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698639.781753:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698639.781754:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698639.781756:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698639.781760:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698639.781770:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698639.781773:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6540 00000100:00080000:1.0:1787698639.781779:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698639.781783:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781786:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781789:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698639.781791:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698639.781794:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781800:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698639.781803:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698639.781808:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698639.781814:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787698639.781821:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698639.781828:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698639.781843:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698639.781844:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698639.781875:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698639.781878:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698639.781879:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698639.781882:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6540 00000100:00080000:1.0:1787698639.781886:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698639.781889:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781892:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781894:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698639.781898:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787698639.781899:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698639/real 1787698639] req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698639.781905:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.781911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.781914:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698639.781918:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698639.781924:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698639.781927:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698639.781929:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698639.781941:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698639.781948:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698639.781956:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698639.781961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698639.781965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698639.781973:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698639.781979:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698639.781985:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787698639.781985:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126363776:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698639.781988:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698639.781993:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126363776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698639.781994:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6540 00000100:00080000:1.0:1787698639.782000:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698639.782007:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698639.782007:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698639.782015:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.782018:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698639.782021:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698639.782026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698639.782027:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698639.782036:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698639.782041:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787698639.782042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363904:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698639.782049:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698639.782057:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698639.782058:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698639.782068:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698639.782069:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698639.782085:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698639.782090:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698639.782094:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698639.782099:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698639.782108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698639/real 1787698639] req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698639.782118:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698639.782146:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6550) 00000100:00100000:2.0:1787698639.782147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698639.782171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698639.782175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698639.782179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782185:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126363904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698639.782189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782195:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698639.782205:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782210:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698639.782218:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698639.782224:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782231:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698639.782234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698639/real 1787698639] req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782246:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698639.782258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698639.782260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698639.782262:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698639.782271:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782276:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698639.782284:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782288:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698639.782294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698639.782297:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782303:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698639.782307:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698639/real 1787698639] req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782324:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698639.782328:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698639.782331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698639.782333:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698639.782341:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126363904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698639.782346:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698639.782350:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126364032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698639.782354:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698639.782357:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126364160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698639.782362:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698639.782367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782372:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698639.782380:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782384:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698639.782390:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698639.782394:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698664 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782400:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698639.782403:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698639/real 1787698639] req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698639.782416:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698639.782455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698639.782458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698639.782460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698639.782465:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698639.782469:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126364288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698664 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698639.782475:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698640.293511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698640.293511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698640.293511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698640.421573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698641.317538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698641.317539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698641.317546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698641.445580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698641.701517:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301215087, at 4301215104 00000100:00000040:2.0:1787698641.701518:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301215087, at 4301215104 00000100:00100000:3.0:1787698642.341498:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698642.341514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698642.341519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698642.469625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698643.365527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698643.365527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698643.365556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698643.493608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698644.389495:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698644.389495:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698644.389497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698644.517696:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698644.901589:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698644.901604:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698644.901609:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698644.901624:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6545 00000100:00080000:1.0:1787698644.901640:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698644.901651:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901656:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901660:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698644.901667:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698644.901671:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901674:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901677:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698644.901681:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901699:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698644.901741:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698644.901762:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698644.901808:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698644.901812:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698644.901815:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698644.901819:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6545 00000100:00080000:1.0:1787698644.901824:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698644.901826:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901829:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901831:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698644.901833:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698644.901836:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901837:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901839:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698644.901841:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698644.901851:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698644.901855:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698644.901864:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698644.901871:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698644.901880:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698644.901884:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698644.901886:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698644.901891:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6545 00000100:00080000:1.0:1787698644.901894:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698644.901898:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901901:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901903:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698644.901906:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698644.901908:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901913:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698644.901916:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698644.901921:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698644.901926:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698644.901936:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698644.901947:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698644.901957:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698644.901960:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698644.901962:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698644.901967:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6545 00000100:00080000:1.0:1787698644.901971:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698644.901974:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901977:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901980:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698644.901982:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698644.901985:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.901989:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.901991:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698644.901994:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698644.902000:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698644.902007:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698644.902017:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698644.902027:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698644.902036:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698644.902039:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698644.902041:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698644.902045:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6545 00000100:00080000:1.0:1787698644.902049:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698644.902052:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.902055:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.902058:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698644.902060:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698644.902063:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698644.902066:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698644.902068:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698644.902071:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698644.902076:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698644.902081:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698644.902090:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698644.902102:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698644.902114:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6556) 00000100:00000040:2.0:1787698644.902215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126364416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698644.902273:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902280:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698644.902325:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698644.902381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902447:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698644/real 1787698644] req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902472:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698644.902479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698644.902482:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698644.902484:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126364416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698644.902495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902501:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698644.902516:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902522:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698644.902532:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698644.902537:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902544:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698644.902548:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698644/real 1787698644] req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902556:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902562:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698644.902567:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698644.902568:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698644.902571:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902575:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698644.902579:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902586:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698644.902599:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902608:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698644.902621:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698644.902628:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902638:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698644.902645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698644/real 1787698644] req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902656:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902666:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698644.902674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698644.902677:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698644.902680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698644.902690:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698644.902707:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902712:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698644.902721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698644.902726:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902733:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698644.902737:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698644/real 1787698644] req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902744:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902750:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698644.902755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698644.902757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698644.902759:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902764:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698644.902767:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698644.902778:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902783:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698644.902788:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698644.902792:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698669 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902802:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698644.902806:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698644/real 1787698644] req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902813:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698644.902818:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698644.902822:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698644.902824:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698644.902826:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698644.902831:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126364928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698644.902836:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126364416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698644.902844:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698644.902852:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126364544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698644.902858:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698644.902861:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126364672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698644.902865:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698644.902869:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126364800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698644.902874:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698644.902877:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126364928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698669 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698644.902882:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698645.413538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698645.413538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698645.413538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698645.541614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698646.437532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698646.437532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698646.437588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698646.565695:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698646.821522:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301220207, at 4301220224 00000100:00000040:2.0:1787698646.821641:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301220207, at 4301220224 00000100:00100000:1.0:1787698647.461553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698647.461553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698647.461557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698647.589554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698648.485489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698648.485496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698648.485501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698648.613590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698649.509478:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698649.509478:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698649.509517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698649.637580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698650.021571:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.021581:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698650.021585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.021594:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6551 00000100:00080000:0.0:1787698650.021603:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698650.021605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021608:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021611:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698650.021615:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698650.021618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021621:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021625:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698650.021629:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021644:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698650.021669:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698650.021683:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698650.021705:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.021709:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698650.021711:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.021717:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6551 00000100:00080000:0.0:1787698650.021721:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698650.021723:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021726:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021728:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698650.021730:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698650.021733:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021736:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698650.021738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698650.021747:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698650.021749:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698650.021752:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698650.021765:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698650.021773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126365056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698650.021778:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698650.021793:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.021798:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698650.021801:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.021806:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6551 00000100:00080000:0.0:1787698650.021811:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698650.021815:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698650.021818:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021834:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698650.021837:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021843:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698650.021846:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698650.021849:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021851:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021852:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698650.021854:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698650.021860:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698650.021864:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698650.021871:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698650.021878:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698650.021886:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698650.021887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.021894:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698650.021896:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.021901:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6551 00000100:00080000:0.0:1787698650.021906:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698650.021913:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698650.021917:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021926:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021933:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698650.021938:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698650.021945:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.021958:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.021960:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698650.021964:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698650.021970:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698650.021976:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698650.021983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698650.021992:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698650.022002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698650.022013:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022025:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698650.022030:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698650.022045:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698650.022050:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698650.022059:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6551 00000100:00080000:2.0:1787698650.022059:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698650.022071:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698650.022073:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698650.022079:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698650.022088:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698650.022090:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698650.022096:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698650.022101:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698650.022112:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698650.022123:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698650.022130:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126365056:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787698650.022132:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698650.022141:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698650.022147:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698650.022149:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126365056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698650.022154:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698650.022172:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787698650.022175:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698650.022195:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698650.022205:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6561) 00000100:00000040:2.0:1787698650.022212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022226:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698650.022245:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022287:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698650.022301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698650.022308:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022319:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698650.022324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.022358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.022361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698650.022366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698650.022380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022389:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698650.022402:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022410:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698650.022448:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698650.022456:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022466:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698650.022471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022483:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022492:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.022502:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.022505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698650.022509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698650.022523:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022532:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698650.022544:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022552:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698650.022563:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698650.022570:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022580:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698650.022585:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.022611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.022614:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698650.022618:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022626:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698650.022632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022641:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698650.022654:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022661:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698650.022672:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698650.022680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698675 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022689:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698650.022694:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698650/real 1787698650] req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022705:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698650.022714:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698650.022721:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698650.022724:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698650.022728:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698650.022736:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698650.022742:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126365184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698650.022749:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698650.022756:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126365312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698650.022763:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698650.022768:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126365440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698650.022775:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698650.022779:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126365568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698675 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698650.022786:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698650.533572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698650.533573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698650.533582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698650.661578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698651.557564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698651.557564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698651.557570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698651.685613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698651.941579:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301225327, at 4301225344 00000100:00000040:3.0:1787698651.941580:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301225327, at 4301225344 00000100:00100000:0.0:1787698652.581572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698652.581572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698652.581602:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698652.709569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698653.605546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698653.605562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698653.605591:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698653.733592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698654.629521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698654.629546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698654.629552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698654.757591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698655.141562:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698655.141571:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698655.141575:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698655.141586:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6556 00000100:00080000:3.0:1787698655.141597:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698655.141601:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.141621:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141624:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698655.141629:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698655.141632:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.141635:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141638:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698655.141641:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141653:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698655.141675:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698655.141689:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698655.141709:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698655.141713:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698655.141715:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698655.141719:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6556 00000100:00080000:3.0:1787698655.141723:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698655.141726:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.141729:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141732:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698655.141735:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698655.141738:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.141741:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141744:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698655.141747:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698655.141756:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698655.141763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698655.141773:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698655.141797:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698655.141798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126365696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698655.141823:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698655.141837:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698655.141840:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698655.141842:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698655.141849:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6556 00000100:00000040:2.0:1787698655.141852:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698655.141856:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698655.141863:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.141870:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141878:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698655.141883:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698655.141893:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.141904:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698655.141906:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141913:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698655.141919:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698655.141925:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698655.141931:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698655.141943:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698655.141953:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698655.141962:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698655.141973:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698655.141976:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698655.141980:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6556 00000100:00080000:3.0:1787698655.141985:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698655.141990:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.141994:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698655.141997:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698655.141999:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698655.142005:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.142008:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698655.142014:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698655.142015:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698655.142026:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698655.142043:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698655.142044:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365824:192.168.201.136@tcp:38:kworker.0 00000020:00000040:3.0:1787698655.142055:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698655.142068:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698655.142076:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698655.142094:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698655.142108:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698655.142114:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787698655.142123:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698655.142128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698655.142131:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698655.142136:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698655.142151:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6556 00000100:00100000:2.0:1787698655.142156:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365952:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698655.142160:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698655.142184:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.142191:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698655.142193:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698655.142207:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698655.142214:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698655.142214:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698655.142224:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698655.142228:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698655.142231:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000020:00000040:3.0:1787698655.142231:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698655.142242:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698655.142248:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698655.142252:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698655.142266:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698655.142271:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366080:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787698655.142285:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698655.142290:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698655.142328:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698655.142332:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698655.142343:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787698655.142352:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6566) 00000100:00000040:2.0:1787698655.142359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698655.142378:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698655.142390:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698655.142398:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698655.142409:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698655.653522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698655.653522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698655.653530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698655.781631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698656.677529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698656.677529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698656.677533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698656.805632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698657.061544:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301230447, at 4301230464 00000100:00000040:3.0:1787698657.061544:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301230447, at 4301230464 00000100:00100000:3.0:1787698657.701540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698657.701541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698657.701541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698657.829599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787698658.213825:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:2.0:1787698658.213888:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787698658.213909:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787698658.213968:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698658.213995:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787698658.214004:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698658.214014:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787698658.214022:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.214033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698658] req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698658.214043:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787698658.214056:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698658.214071:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787698658.214084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698658.214085:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698680 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698658.214748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.214763:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.214769:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698658.214777:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.214786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365824:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698658.214804:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698658] req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.214820:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698658.214828:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.214836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.214839:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698658.214843:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.214850:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126365952:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698658.214859:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698658] req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.214872:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698658.214880:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.214887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.214890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698658.214893:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.214900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126365696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698658.214907:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126365824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698658.214920:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698658.214932:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126365952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698658.214939:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698658.214946:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126365696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698658.214954:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698658.214964:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698658] req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.214977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698658.214986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.215006:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.215010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698658.215014:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.215021:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366208:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698658.215027:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698655/real 1787698658] req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.215038:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698658.215046:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698658.215053:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698658.215056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698658.215059:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698658.215067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698658.215072:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126366208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698658.215080:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698658.215087:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126366080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698680 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698658.215094:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698658.725550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698658.725551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698658.725554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698658.853628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698659.749539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698659.749539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698659.749544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698659.877554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698660.261570:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698660.261584:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698660.261589:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698660.261602:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6561 00000100:00080000:2.0:1787698660.261616:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698660.261621:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261630:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698660.261636:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698660.261639:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261641:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261645:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698660.261660:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261674:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698660.261708:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698660.261735:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698660.261764:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698660.261769:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698660.261771:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698660.261775:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6561 00000100:00080000:2.0:1787698660.261779:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698660.261783:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261786:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261788:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698660.261791:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698660.261794:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261797:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261799:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698660.261802:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698660.261813:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698660.261819:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698660.261830:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698660.261839:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698660.261847:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698660.261851:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698660.261853:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698660.261856:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6561 00000100:00080000:2.0:1787698660.261859:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698660.261862:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261864:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261867:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698660.261869:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698660.261872:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261888:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698660.261891:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698660.261896:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698660.261900:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698660.261911:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698660.261920:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698660.261928:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698660.261932:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698660.261944:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698660.261949:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6561 00000100:00080000:2.0:1787698660.261952:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698660.261955:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261958:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261960:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698660.261962:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698660.261965:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.261968:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.261970:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698660.261972:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698660.261978:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698660.261982:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698660.261991:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698660.262000:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698660.262009:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698660.262012:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698660.262013:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698660.262017:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6561 00000100:00080000:2.0:1787698660.262021:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698660.262024:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.262026:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.262028:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698660.262040:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698660.262044:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698660.262046:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698660.262049:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698660.262051:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698660.262056:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698660.262060:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698660.262068:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698660.262089:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698660.262101:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6571) 00000100:00000040:1.0:1787698660.262284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262301:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126366336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698660.262338:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262347:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698660.262404:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698660.262457:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698660/real 1787698660] req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698660.262535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698660.262540:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698660.262545:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262554:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126366336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698660.262561:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262570:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698660.262585:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262593:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698660.262604:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698660.262611:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262622:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698660.262628:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698660/real 1787698660] req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262638:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262661:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698660.262669:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698660.262672:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698660.262676:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262683:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698660.262689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698660.262709:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262716:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698660.262727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698660.262733:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262741:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698660.262747:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698660/real 1787698660] req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262765:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698660.262773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698660.262777:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698660.262780:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262788:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698660.262793:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262802:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698660.262814:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262821:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698660.262831:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698660.262838:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262846:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698660.262851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698660/real 1787698660] req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262862:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262870:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698660.262876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698660.262879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698660.262882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262888:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698660.262893:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262901:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698660.262914:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262921:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698660.262931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698660.262937:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698685 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262946:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698660.262952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698660/real 1787698660] req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698660.262970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698660.262977:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698660.262980:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698660.262984:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698660.262991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126366848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698660.262998:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126366336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698660.263008:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698660.263026:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126366464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698660.263039:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698660.263045:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126366592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698660.263052:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698660.263057:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126366720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698660.263065:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698660.263069:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126366848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698685 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698660.263088:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698660.773521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698660.773521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698660.773553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698660.901639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698661.797521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698661.797523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698661.797542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698661.925608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698662.181541:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301235568, at 4301235584 00000100:00000040:2.0:1787698662.181564:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301235568, at 4301235584 00000100:00100000:0.0:1787698662.821537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698662.821537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698662.821571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698662.949663:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698663.845485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698663.845491:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698663.845496:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698663.973639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698664.869541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698664.869546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698664.869546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698664.997550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698665.382126:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698665.382139:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698665.382144:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698665.382157:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6566 00000100:00080000:2.0:1787698665.382170:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698665.382174:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382178:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382182:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698665.382188:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698665.382191:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382194:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382198:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698665.382201:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382220:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698665.382251:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698665.382273:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698665.382296:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698665.382301:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698665.382303:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698665.382307:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6566 00000100:00080000:2.0:1787698665.382311:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698665.382314:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382316:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382320:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698665.382322:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698665.382325:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382327:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382329:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698665.382331:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698665.382340:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698665.382340:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698665.382345:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698665.382358:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698665.382359:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126366976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698665.382372:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698665.382378:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698665.382381:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698665.382382:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698665.382385:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6566 00000100:00080000:2.0:1787698665.382388:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698665.382389:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382391:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382393:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787698665.382397:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698665.382397:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698665.382408:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382410:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382413:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382414:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698665.382416:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698665.382436:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698665.382440:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698665.382449:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698665.382456:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698665.382461:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698665.382463:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698665.382465:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698665.382469:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6566 00000100:00080000:2.0:1787698665.382473:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698665.382476:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382479:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382481:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698665.382484:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698665.382487:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382490:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698665.382500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382502:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698665.382507:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698665.382514:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698665.382518:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698665.382527:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698665.382528:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698665.382550:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698665.382555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698665.382558:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698665.382560:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698665.382562:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6566 00000100:00080000:2.0:1787698665.382565:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698665.382567:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382575:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382578:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698665.382583:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698665.382587:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698665.382589:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698665.382592:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698665.382597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698665/real 1787698665] req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698665.382597:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698665.382605:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698665.382614:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698665.382627:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698665.382627:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698665.382649:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698665.382661:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6576) 00000100:00000040:1.0:1787698665.382662:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698665.382674:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698665.382679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698665.382684:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126366976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698665.382701:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126366976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698665.382709:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698665.382728:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698665.382759:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382764:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698665.382774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698665.382780:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382787:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698665.382791:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698665/real 1787698665] req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382799:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382805:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698665.382810:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698665.382811:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698665.382813:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382817:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698665.382821:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698665.382842:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382846:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698665.382853:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698665.382857:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382863:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698665.382866:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698665/real 1787698665] req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382877:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698665.382888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698665.382889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698665.382891:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382896:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698665.382899:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382904:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698665.382911:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382916:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698665.382925:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698665.382930:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382940:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698665.382945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698665/real 1787698665] req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382961:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.382969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698665.382975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698665.382978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698665.382981:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.382988:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698665.383005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.383013:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698665.383025:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.383030:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698665.383041:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698665.383047:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698690 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.383055:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698665.383059:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698665/real 1787698665] req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.383068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698665.383076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698665.383083:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698665.383091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698665.383095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698665.383101:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698665.383107:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126367104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698665.383114:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698665.383120:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126367232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698665.383127:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698665.383132:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126367360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698665.383138:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698665.383142:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126367488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698690 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698665.383148:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698665.893539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698665.893540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698665.893540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698666.021589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698666.917528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698666.917528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698666.917528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698667.045638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698667.301566:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301240688, at 4301240704 00000100:00000040:2.0:1787698667.301577:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301240688, at 4301240704 00000100:00100000:0.0:1787698667.941555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698667.941555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698667.941565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698668.069650:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698668.965531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698668.965543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698668.965544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698669.093653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698669.989552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698669.989552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698669.989554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698670.117639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698670.501940:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698670.501949:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698670.501954:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698670.501963:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6571 00000100:00080000:0.0:1787698670.501984:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698670.501988:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.501993:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.501997:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698670.502003:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698670.502006:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502008:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502012:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698670.502015:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502031:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698670.502060:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698670.502072:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698670.502099:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698670.502105:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698670.502107:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698670.502112:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6571 00000100:00080000:0.0:1787698670.502116:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698670.502119:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502121:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502123:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698670.502126:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698670.502129:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502132:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502134:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698670.502136:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698670.502144:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698670.502151:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698670.502156:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698670.502159:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126367616:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698670.502165:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698670.502174:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698670.502182:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698670.502184:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698670.502187:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698670.502190:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698670.502191:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6571 00000100:00080000:0.0:1787698670.502195:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698670.502197:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502200:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698670.502200:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502202:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698670.502205:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698670.502207:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502209:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502211:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698670.502213:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698670.502217:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698670.502220:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698670.502235:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698670.502243:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698670.502243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698670.502250:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698670.502253:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698670.502255:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698670.502258:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6571 00000100:00080000:0.0:1787698670.502261:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698670.502262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698670.502264:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502267:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502269:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698670.502271:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698670.502274:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502276:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502277:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698670.502279:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698670.502290:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698670.502294:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698670.502295:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698670/real 1787698670] req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698670.502301:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698670.502309:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502312:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698670.502317:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698670.502320:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698670.502321:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698670.502322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698670.502325:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6571 00000100:00080000:0.0:1787698670.502328:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698670.502330:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698670.502331:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787698670.502333:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502335:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:1.0:1787698670.502335:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:0.0:1787698670.502337:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698670.502340:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698670.502340:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698670.502343:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698670.502345:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698670.502347:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698670.502351:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698670.502355:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698670.502357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126367616:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698670.502362:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698670.502365:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126367616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698670.502371:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698670.502377:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698670.502380:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6581) 00000100:00000040:1.0:1787698670.502395:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502404:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698670.502442:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502447:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698670.502457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698670.502462:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502469:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698670.502473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698670/real 1787698670] req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502480:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698670.502490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698670.502492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698670.502494:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698670.502503:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698670.502522:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502529:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698670.502540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698670.502546:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502552:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698670.502555:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698670/real 1787698670] req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502568:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698670.502573:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698670.502574:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698670.502576:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126367872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698670.502583:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698670.502598:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502602:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698670.502609:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698670.502629:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502640:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698670.502645:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698670/real 1787698670] req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502665:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698670.502673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698670.502676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698670.502679:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502684:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698670.502688:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698670.502708:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502716:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698670.502727:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698670.502734:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698695 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502743:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698670.502749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698670/real 1787698670] req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502760:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698670.502769:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698670.502776:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698670.502779:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698670.502783:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698670.502790:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698670.502796:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126367744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698670.502814:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698670.502820:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126367872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698670.502834:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698670.502839:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126368000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698670.502854:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698670.502859:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126368128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698695 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698670.502866:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698671.013551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698671.013551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698671.013551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698671.141623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698672.037519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698672.037519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698672.037536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698672.165618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698672.421569:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301245807, at 4301245824 00000100:00000040:2.0:1787698672.421572:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301245807, at 4301245824 00000100:00100000:2.0:1787698673.061579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698673.061631:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698673.061645:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698673.189634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698674.085521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698674.085521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698674.085530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698674.213649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698675.109557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698675.109557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698675.109591:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698675.237619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698675.621565:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698675.621579:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698675.621585:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698675.621598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6576 00000100:00080000:1.0:1787698675.621611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698675.621616:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621621:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621625:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698675.621631:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698675.621635:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621639:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621643:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698675.621655:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621676:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698675.621705:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698675.621718:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.621738:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698675.621740:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698675.621741:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698675.621744:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6576 00000100:00080000:1.0:1787698675.621747:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698675.621749:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621751:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621752:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698675.621755:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698675.621757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621760:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621761:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698675.621764:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698675.621773:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698675.621774:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698675.621777:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698675.621792:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698675.621798:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126368256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698675.621812:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.621849:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698675.621853:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00000040:2.0:1787698675.621856:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.621866:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698675.621868:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698675.621875:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6576 00000100:00080000:1.0:1787698675.621879:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698675.621881:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621883:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621885:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698675.621887:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698675.621889:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621891:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621893:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698675.621894:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698675.621898:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698675.621901:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698675.621910:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698675.621919:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621920:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.621928:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698675.621931:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698675.621933:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698675.621939:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6576 00000100:00000040:2.0:1787698675.621943:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.621944:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698675.621953:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621957:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621960:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698675.621963:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698675.621966:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.621969:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.621972:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698675.621975:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698675.621980:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698675.621981:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698675/real 1787698675] req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698675.621985:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698675.621997:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698675.622008:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.622020:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698675.622026:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698675.622029:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698675.622035:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6576 00000100:00000040:2.0:1787698675.622036:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.622038:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698675.622042:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.622046:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698675.622051:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698675.622055:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:2.0:1787698675.622060:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:1.0:1787698675.622061:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698675.622068:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698675.622071:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787698675.622073:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787698675.622079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698675.622080:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698675.622088:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698675.622096:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698675.622098:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126368256:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698675.622109:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698675.622111:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126368256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698675.622136:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698675.622137:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698675.622150:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6586) 00000100:00000040:2.0:1787698675.622169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698675.622195:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622202:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698675.622215:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698675.622222:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622233:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698675.622239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698675/real 1787698675] req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622248:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622255:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698675.622260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698675.622261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698675.622264:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622268:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698675.622272:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698675.622288:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622295:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698675.622306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698675.622313:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622322:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698675.622328:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698675/real 1787698675] req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622354:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698675.622362:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698675.622365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698675.622369:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622376:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698675.622382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698675.622410:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622442:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698675.622460:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698675.622467:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622476:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698675.622482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698675/real 1787698675] req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622494:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698675.622508:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698675.622511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698675.622515:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622523:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698675.622528:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622536:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698675.622550:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622558:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698675.622569:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698675.622576:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698700 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622582:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698675.622586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698675/real 1787698675] req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622593:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698675.622598:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698675.622602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698675.622604:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698675.622606:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698675.622610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126368768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698675.622613:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126368384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698675.622619:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698675.622623:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126368512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698675.622627:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698675.622630:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126368640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698675.622634:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698675.622637:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126368768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698700 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698675.622642:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698676.133492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698676.133498:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698676.133598:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698676.261646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698677.157481:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698677.157494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698677.157506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698677.285623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698677.541546:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301250927, at 4301250944 00000100:00000040:3.0:1787698677.541546:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301250927, at 4301250944 00000100:00100000:3.0:1787698678.181496:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698678.181516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698678.181568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698678.309637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698679.205553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698679.205553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698679.205553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698679.333605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698680.229555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698680.229555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698680.229555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698680.357658:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698680.741530:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698680.741543:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698680.741548:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698680.741559:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6581 00000100:00080000:0.0:1787698680.741571:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698680.741573:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741576:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741580:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698680.741584:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698680.741586:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741588:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741591:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698680.741593:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741607:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698680.741632:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698680.741645:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698680.741664:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698680.741667:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698680.741668:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698680.741671:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6581 00000100:00080000:0.0:1787698680.741674:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698680.741675:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741677:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741678:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698680.741680:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698680.741682:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741683:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741685:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698680.741693:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698680.741704:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698680.741707:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698680.741714:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698680.741717:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698680.741744:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698680.741760:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698680.741763:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787698680.741764:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126368896:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698680.741766:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698680.741773:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6581 00000100:00080000:0.0:1787698680.741776:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698680.741778:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741779:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741781:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698680.741783:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698680.741785:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741787:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741788:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698680.741790:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698680.741793:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698680.741797:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698680.741798:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698680.741808:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698680.741811:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698680.741818:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698680.741823:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698680.741825:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698680.741826:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698680.741829:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6581 00000100:00080000:0.0:1787698680.741831:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698680.741832:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741834:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741835:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698680.741837:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698680.741838:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741839:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741841:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698680.741843:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698680.741846:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698680.741848:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741853:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698680.741861:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698680.741867:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698680.741875:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698680.741882:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698680.741884:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698680.741885:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698680.741887:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6581 00000100:00080000:0.0:1787698680.741889:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698680.741890:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741893:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698680.741895:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698680.741896:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698680.741898:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698680.741900:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698680.741901:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698680.741904:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698680.741906:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698680/real 1787698680] req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698680.741908:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698680.741917:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698680.741924:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.741929:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698680.741934:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6591) 00000100:00000040:2.0:1787698680.741938:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698680.741945:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698680.741949:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698680.741953:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.741966:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126368896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698680.741972:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126368896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698680.741980:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698680.741993:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.741999:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698680.742013:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742020:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698680.742033:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698680.742040:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742051:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698680.742057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698680/real 1787698680] req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742067:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698680.742089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698680.742093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698680.742097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742103:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698680.742106:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742112:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698680.742124:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742131:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698680.742142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698680.742149:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742159:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698680.742165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698680/real 1787698680] req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742176:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742185:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698680.742193:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698680.742196:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698680.742198:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698680.742206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742212:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698680.742222:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742226:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698680.742234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698680.742239:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742248:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698680.742253:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698680/real 1787698680] req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698680.742280:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698680.742283:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698680.742287:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698680.742297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742302:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698680.742311:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742315:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698680.742322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698680.742327:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698705 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742332:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698680.742335:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698680/real 1787698680] req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698680.742347:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698680.742354:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698680.742357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698680.742361:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698680.742368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698680.742374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126369024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698680.742382:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698680.742389:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126369152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698680.742395:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698680.742399:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126369280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698680.742403:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698680.742406:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126369408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698705 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698680.742409:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698681.253510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698681.253510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698681.253516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698681.381623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698682.277528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698682.277528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698682.277555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698682.405613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698682.661538:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301256047, at 4301256064 00000100:00000040:3.0:1787698682.661538:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301256047, at 4301256064 00000100:00100000:0.0:1787698683.301486:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698683.301486:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698683.301487:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698683.429616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698684.325544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698684.325544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698684.325550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698684.453582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698685.349553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698685.349553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698685.349553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698685.477641:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698685.861573:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698685.861585:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698685.861591:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698685.861604:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6586 00000100:00080000:1.0:1787698685.861617:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698685.861622:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698685.861627:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.861650:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698685.861656:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698685.861661:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698685.861664:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.861668:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698685.861671:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698685.861690:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698685.861724:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698685.861741:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698685.861764:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698685.861769:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698685.861770:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698685.861775:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6586 00000100:00080000:1.0:1787698685.861778:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698685.861781:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698685.861784:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.861786:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698685.861790:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698685.861794:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698685.861797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.861799:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698685.861802:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698685.861810:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698685.861815:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698685.861824:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698685.861825:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126369536:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698685.861844:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698685.861856:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.861858:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.861885:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698685.861888:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698685.861892:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698685.861893:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698685.861897:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6586 00000100:00080000:1.0:1787698685.861902:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698685.861904:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698685.861906:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.861907:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698685.861909:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698685.861912:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698685.861914:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.861916:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698685.861919:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:2.0:1787698685.861924:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:1.0:1787698685.861925:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698685.861932:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698685.861946:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698685.861947:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698685.861972:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698685.861983:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698685.861989:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787698685.861989:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698685/real 1787698685] req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698685.861993:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698685.862007:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6586 00000100:00080000:1.0:1787698685.862012:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698685.862016:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698685.862018:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698685.862025:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.862035:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:2.0:1787698685.862037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698685.862041:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698685.862049:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698685.862049:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698685.862056:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698685.862057:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.862063:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698685.862064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698685.862071:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698685.862076:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126369536:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787698685.862081:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698685.862084:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126369536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698685.862088:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:2.0:1787698685.862094:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698685.862102:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698685.862112:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698685.862117:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862122:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369664:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698685.862136:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698685.862141:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698685.862142:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698685.862146:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698685.862151:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6586 00000100:00080000:1.0:1787698685.862156:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698685.862163:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698685.862164:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698685.862170:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.862174:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698685.862177:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:2.0:1787698685.862178:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698685.862183:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698685.862188:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698685.862190:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698685.862196:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698685.862202:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698685.862204:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000008:00100000:1.0:1787698685.862209:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698685.862212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698685/real 1787698685] req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698685.862217:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698685.862222:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698685.862230:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698685.862231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698685.862257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698685.862264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698685.862265:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698685.862300:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6596) 00000100:00000040:2.0:1787698685.862301:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862312:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698685.862324:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862333:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698685.862345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698685.862352:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862359:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698685.862363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698685/real 1787698685] req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698685.862380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698685.862382:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698685.862384:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862388:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698685.862392:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126369664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698685.862397:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698685.862400:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126369792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698685.862404:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698685.862409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862415:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698685.862445:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862453:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698685.862461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698685.862465:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862471:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698685.862476:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698685/real 1787698685] req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862494:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698685.862512:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698685.862515:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698685.862519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126369920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698685.862532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698685.862552:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862559:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698685.862570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698685.862575:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698710 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862588:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698685.862594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698685/real 1787698685] req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698685.862613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698685.862621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698685.862624:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698685.862628:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698685.862635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698685.862648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126369920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698685.862656:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698685.862660:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126370048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698710 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698685.862664:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698686.373530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698686.373530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698686.373536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698686.501550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698687.397548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698687.397553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698687.397554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698687.525646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698687.781514:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301261167, at 4301261184 00000100:00000040:3.0:1787698687.781514:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301261167, at 4301261184 00000100:00100000:0.0:1787698688.421557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698688.421557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698688.421557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698688.549629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698689.445525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698689.445525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698689.445531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698689.573649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698690.469552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698690.469552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698690.469552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698690.597657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698690.981546:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698690.981565:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698690.981571:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698690.981584:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6591 00000100:00080000:2.0:1787698690.981596:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698690.981601:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981605:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981608:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698690.981613:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698690.981615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981617:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981619:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698690.981622:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981637:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698690.981661:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698690.981673:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698690.981693:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698690.981695:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698690.981697:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698690.981699:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6591 00000100:00080000:2.0:1787698690.981702:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698690.981705:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981707:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981709:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698690.981712:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698690.981715:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981716:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981718:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698690.981719:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698690.981727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698690.981728:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698690.981734:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698690.981744:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698690.981747:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126370176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698690.981758:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698690.981763:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698690.981765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698690.981766:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698690.981768:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6591 00000100:00080000:2.0:1787698690.981770:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698690.981771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981774:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698690.981775:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698690.981777:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698690.981782:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698690.981782:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981787:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698690.981792:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698690.981799:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698690.981800:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981806:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698690.981813:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698690.981818:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698690.981824:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698690.981826:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698690.981827:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698690.981830:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6591 00000100:00080000:2.0:1787698690.981832:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698690.981834:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981835:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981836:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698690.981837:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698690.981839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981841:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698690.981845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981845:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698690.981851:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698690.981855:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698690.981859:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698690.981864:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698690.981867:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698690.981884:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698690.981889:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698690.981892:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698690.981893:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698690.981895:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6591 00000100:00080000:2.0:1787698690.981897:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698690.981900:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698690.981901:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981903:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698690.981904:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698690.981908:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698690.981909:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698690/real 1787698690] req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698690.981910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698690.981913:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698690.981916:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698690.981921:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698690.981926:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698690.981933:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698690.981940:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.981959:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698690.981962:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698690.981974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698690.981979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698690.981981:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6602) 00000100:00000040:1.0:1787698690.981985:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.981992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126370176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698690.981997:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126370176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698690.982005:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698690.982019:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982024:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698690.982036:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982041:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698690.982049:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698690.982054:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982061:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698690.982064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698690/real 1787698690] req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698690.982081:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698690.982083:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698690.982085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982089:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698690.982092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698690.982111:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982119:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698690.982129:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698690.982135:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982144:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698690.982150:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698690/real 1787698690] req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982162:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698690.982167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698690.982169:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698690.982171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982175:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698690.982179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698690.982192:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982197:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698690.982203:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698690.982207:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982213:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698690.982216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698690/real 1787698690] req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698690.982237:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698690.982239:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698690.982241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698690.982247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698690.982257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982261:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698690.982274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698690.982278:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698715 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982283:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698690.982286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698690/real 1787698690] req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698690.982298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698690.982302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698690.982304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698690.982306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698690.982310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698690.982312:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126370304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698690.982317:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698690.982320:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126370432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698690.982324:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698690.982327:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126370560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698690.982331:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698690.982333:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126370688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698715 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698690.982338:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698691.493522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698691.493522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698691.493522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698691.621629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698692.517551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698692.517551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698692.517552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698692.645604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698692.901555:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301266287, at 4301266304 00000100:00000040:3.0:1787698692.901555:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301266287, at 4301266304 00000100:00100000:1.0:1787698693.541494:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698693.541494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698693.541499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698693.669641:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698694.565548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698694.565548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698694.565552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698694.693574:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698695.589560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698695.589560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698695.589560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698695.717637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698696.101558:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698696.101572:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698696.101577:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698696.101591:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6597 00000100:00080000:2.0:1787698696.101606:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698696.101610:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101615:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101619:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698696.101625:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698696.101629:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101632:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101635:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698696.101639:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101657:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698696.101690:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698696.101707:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.101734:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698696.101737:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698696.101739:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698696.101744:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6597 00000100:00080000:2.0:1787698696.101748:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698696.101751:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101754:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101756:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698696.101759:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698696.101762:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101767:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101769:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698696.101772:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698696.101783:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698696.101787:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698696.101792:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698696.101809:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698696.101810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126370816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698696.101827:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.101836:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698696.101840:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698696.101842:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698696.101845:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6597 00000100:00080000:2.0:1787698696.101850:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698696.101853:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698696.101856:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101867:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698696.101871:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698696.101875:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101884:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698696.101887:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698696.101892:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698696.101896:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698696.101907:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698696.101916:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101918:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.101929:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698696.101933:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698696.101936:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698696.101937:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.101946:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6597 00000100:00080000:2.0:1787698696.101953:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698696.101956:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101960:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.101962:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698696.101964:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698696.101968:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.101972:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698696.101974:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698696.101977:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698696.101985:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698696.101991:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698696.101993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698696.101998:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698696.102003:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.102012:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698696.102017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698696.102024:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698696.102030:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102031:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126370816:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698696.102059:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698696.102067:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698696.102070:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698696.102071:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126370816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698696.102076:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6597 00000100:00080000:2.0:1787698696.102083:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000020:00000040:1.0:1787698696.102084:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698696.102088:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698696.102092:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.102096:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698696.102098:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698696.102101:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698696.102104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698696.102107:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698696.102113:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698696.102116:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370944:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698696.102118:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698696.102124:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698696.102131:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698696.102136:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698696.102143:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698696.102144:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698696.102156:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698696.102158:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102163:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102171:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698696.102176:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698696.102194:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6607) 00000100:00000040:1.0:1787698696.102206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698696.102214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698696.102217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698696.102222:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126370944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698696.102242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698696.102259:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102266:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698696.102277:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698696.102285:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102294:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698696.102299:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698696.102319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698696.102320:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698696.102323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102327:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698696.102331:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102337:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698696.102345:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102350:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698696.102357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698696.102361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102367:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698696.102370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698696.102386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698696.102387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698696.102389:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698696.102396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126370944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698696.102401:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698696.102406:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126371072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698696.102410:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698696.102413:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126371200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698696.102417:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698696.102444:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698696.102456:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102460:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698696.102481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698696.102486:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698721 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102492:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698696.102495:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698696/real 1787698696] req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102502:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698696.102508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698696.102516:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698696.102518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698696.102521:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698696.102525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698696.102528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126371328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698721 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698696.102533:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698696.613537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698696.613537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698696.613566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698696.741642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698697.637581:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698697.637581:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698697.637581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698697.765598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698698.021553:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301271407, at 4301271424 00000100:00000040:3.0:1787698698.021553:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301271407, at 4301271424 00000100:00100000:1.0:1787698698.661512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698698.661512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698698.661544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698698.789648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698699.685545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698699.685553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698699.685554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698699.813558:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698700.709498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698700.709498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698700.709499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698700.837633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698701.221565:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698701.221576:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698701.221582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698701.221595:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6602 00000100:00080000:2.0:1787698701.221607:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698701.221610:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221616:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698701.221621:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698701.221625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221626:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221629:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698701.221631:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221646:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698701.221674:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698701.221693:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698701.221714:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698701.221716:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698701.221718:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698701.221721:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6602 00000100:00080000:2.0:1787698701.221723:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698701.221725:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221726:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221728:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698701.221730:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698701.221731:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221733:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221734:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698701.221736:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698701.221745:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698701.221748:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698701.221755:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698701.221764:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698701.221786:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.221792:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126371456:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698701.221795:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698701.221799:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698701.221800:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698701.221803:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6602 00000100:00080000:2.0:1787698701.221805:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698701.221807:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221809:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698701.221811:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698701.221813:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221814:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221815:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698701.221817:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698701.221820:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698701.221823:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698701.221825:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698701.221832:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698701.221839:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221848:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698701.221857:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698701.221861:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698701.221863:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698701.221868:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6602 00000100:00080000:2.0:1787698701.221874:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698701.221877:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221880:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221882:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000800:00000100:1.0:1787698701.221885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221886:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698701.221891:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221894:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221897:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698701.221900:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698701.221904:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698701.221905:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698701.221915:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698701.221928:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698701.221938:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.221945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698701/real 1787698701] req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698701.221949:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698701.221958:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698701.221961:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698701.221967:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6602 00000100:00080000:2.0:1787698701.221971:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698701.221974:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698701.221978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221983:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221987:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698701.221990:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698701.221993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698701.221994:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698701.221996:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698701.221999:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698701.222000:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698701.222006:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698701.222010:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:2.0:1787698701.222011:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698701.222018:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:2.0:1787698701.222022:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698701.222028:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698701.222033:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126371456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698701.222053:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6612) 00000100:00000040:1.0:1787698701.222064:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126371456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698701.222075:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698701.222092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698701.222115:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222122:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698701.222134:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698701.222142:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222149:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698701.222153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698701/real 1787698701] req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698701.222171:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698701.222173:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698701.222175:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222179:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698701.222182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698701.222197:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222204:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698701.222214:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698701.222220:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222235:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698701.222241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698701/real 1787698701] req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222261:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698701.222270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698701.222273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698701.222278:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698701.222292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698701.222314:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222322:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698701.222331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698701.222337:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222347:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698701.222352:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698701/real 1787698701] req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698701.222391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698701.222395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698701.222399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698701.222410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222417:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698701.222459:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222466:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698701.222477:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698701.222484:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698726 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222493:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698701.222499:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698701/real 1787698701] req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222517:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698701.222525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698701.222533:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698701.222536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698701.222539:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698701.222547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126371968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698701.222553:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126371584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698701.222561:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698701.222567:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126371712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698701.222574:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698701.222579:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126371840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698701.222587:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698701.222592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126371968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698726 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698701.222596:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698701.733560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698701.733560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698701.733567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698701.861601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698702.757542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698702.757542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698702.757591:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698702.885591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698703.141472:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301276527, at 4301276544 00000100:00000040:2.0:1787698703.141531:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301276527, at 4301276544 00000100:00100000:3.0:1787698703.781503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698703.781506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698703.781560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698703.909656:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698704.805537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698704.805537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698704.805538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698704.933654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698705.829505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698705.829505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698705.829513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698705.957553:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698706.341567:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698706.341578:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698706.341582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698706.341592:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6607 00000100:00080000:2.0:1787698706.341603:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698706.341605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341608:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341611:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698706.341616:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698706.341618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341620:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341623:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698706.341625:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341640:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698706.341667:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698706.341681:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698706.341707:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698706.341711:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698706.341713:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698706.341717:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6607 00000100:00080000:2.0:1787698706.341729:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698706.341736:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341741:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341743:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698706.341747:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698706.341750:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341754:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341756:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698706.341759:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698706.341765:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698706.341773:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698706.341780:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698706.341784:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126372096:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698706.341795:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698706.341808:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698706.341819:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698706.341823:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698706.341825:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698706.341829:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698706.341838:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6607 00000100:00000040:1.0:1787698706.341842:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698706.341847:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698706.341853:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341856:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341858:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698706.341861:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698706.341864:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341866:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341869:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698706.341872:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698706.341877:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698706.341882:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698706.341890:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698706.341893:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698706.341904:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.341913:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698706.341915:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698706.341923:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698706.341927:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698706.341935:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6607 00000100:00080000:2.0:1787698706.341939:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698706.341942:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341944:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341947:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698706.341949:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698706.341951:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.341953:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.341955:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698706.341957:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698706.341961:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698706.341962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698706/real 1787698706] req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698706.341966:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698706.341975:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698706.341990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698706.341990:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698706.342019:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698706.342022:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698706.342024:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698706.342026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698706.342029:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6607 00000100:00080000:2.0:1787698706.342035:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698706.342042:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698706.342046:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787698706.342047:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.342052:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:1.0:1787698706.342055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:2.0:1787698706.342056:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698706.342062:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698706.342063:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698706.342071:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698706.342077:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698706.342084:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126372096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698706.342092:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698706.342094:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126372096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698706.342100:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698706.342107:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:2.0:1787698706.342107:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698706.342122:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698706.342130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698706.342136:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698706.342163:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6617) 00000100:00000040:1.0:1787698706.342172:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342182:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698706.342195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698706.342203:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342214:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698706.342219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698706/real 1787698706] req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342229:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698706.342244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698706.342248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698706.342252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698706.342275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698706.342328:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342336:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698706.342347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698706.342354:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342363:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698706.342368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698706/real 1787698706] req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342378:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342386:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698706.342394:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698706.342397:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698706.342401:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698706.342447:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342456:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698706.342469:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342476:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698706.342494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698706.342501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342509:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698706.342515:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698706/real 1787698706] req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342525:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698706.342539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698706.342542:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698706.342546:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342553:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698706.342559:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126372224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698706.342566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698706.342571:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126372352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698706.342578:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698706.342583:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126372480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698706.342590:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698706.342596:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698706.342615:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342621:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698706.342631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698706.342637:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698731 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342645:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698706.342650:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698706/real 1787698706] req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342660:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698706.342668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698706.342675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698706.342678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698706.342682:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698706.342689:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698706.342694:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126372608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698731 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698706.342702:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698706.853478:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698706.853483:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698706.853491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698706.981636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698707.877545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698707.877545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698707.877545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698708.005646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698708.261510:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301281647, at 4301281664 00000100:00000040:2.0:1787698708.261595:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301281647, at 4301281664 00000100:00100000:0.0:1787698708.901531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698708.901531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698708.901531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698709.029602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698709.925534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698709.925534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698709.925534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698710.053599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698710.949539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698710.949540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698710.949580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698711.077589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698711.461470:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698711.461478:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698711.461481:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698711.461489:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6612 00000100:00080000:2.0:1787698711.461500:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698711.461503:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461507:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461511:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698711.461516:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698711.461520:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461523:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461527:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698711.461530:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461545:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698711.461565:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698711.461575:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698711.461598:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698711.461602:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698711.461604:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698711.461608:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6612 00000100:00080000:2.0:1787698711.461612:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698711.461615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461618:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461620:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698711.461623:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698711.461625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461628:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461629:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698711.461631:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698711.461638:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698711.461642:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698711.461654:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698711.461665:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.461681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698711.461686:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698711.461697:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698711.461700:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698711.461715:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6612 00000100:00080000:2.0:1787698711.461724:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698711.461726:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461727:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461730:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00100000:1.0:1787698711.461733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126372736:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787698711.461733:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698711.461744:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461749:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461751:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698711.461753:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698711.461769:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698711.461775:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698711.461783:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698711.461787:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698711.461844:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.461848:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698711.461859:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698711.461862:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698711.461863:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698711.461866:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6612 00000100:00080000:2.0:1787698711.461870:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698711.461873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461876:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461878:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698711.461880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698711.461883:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461886:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461888:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698711.461893:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698711.461898:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698711.461905:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000800:00000100:1.0:1787698711.461907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698711.461919:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698711.461929:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.461937:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698711.461939:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698711.461951:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698711.461955:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698711.461969:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6612 00000100:00080000:2.0:1787698711.461977:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698711.461980:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461987:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698711.461989:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698711.461992:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698711.461995:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698711.461997:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698711.462001:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698711.462006:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698711.462011:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698711.462016:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698711/real 1787698711] req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698711.462028:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698711.462044:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462047:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462069:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698711.462088:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6622) 00000100:00080000:1.0:1787698711.462091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698711.462098:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698711.462104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126372736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698711.462123:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126372736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698711.462135:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698711.462157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698711.462184:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462192:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698711.462223:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698711.462231:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462249:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698711.462257:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698711/real 1787698711] req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462276:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698711.462283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698711.462286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698711.462290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462298:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698711.462305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698711.462329:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462339:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698711.462351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698711.462358:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462367:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698711.462373:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698711/real 1787698711] req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462383:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698711.462400:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698711.462403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698711.462406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126372992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698711.462481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462491:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698711.462505:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462513:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698711.462524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698711.462529:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462536:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698711.462540:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698711/real 1787698711] req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462551:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462557:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698711.462561:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698711.462563:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698711.462565:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698711.462573:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698711.462586:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462590:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698711.462596:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698711.462600:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698736 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462605:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698711.462608:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698711/real 1787698711] req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698711.462620:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698711.462624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698711.462626:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698711.462628:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698711.462631:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698711.462634:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126372864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698711.462638:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698711.462643:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126372992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698711.462650:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698711.462655:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126373120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698711.462661:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698711.462665:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126373248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698736 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698711.462673:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698711.973558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698711.973562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698711.973568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698712.101583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698712.997531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698712.997549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698712.997566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698713.125617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698713.381546:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301286767, at 4301286784 00000100:00000040:3.0:1787698713.381546:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301286767, at 4301286784 00000100:00100000:3.0:1787698714.021514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698714.021515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698714.021515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698714.149536:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698715.045489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698715.045493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698715.045579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698715.173615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698716.069552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698716.069552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698716.069559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698716.197608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698716.581568:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698716.581578:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698716.581582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698716.581592:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6617 00000100:00080000:2.0:1787698716.581603:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698716.581606:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581612:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698716.581616:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698716.581618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581620:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581622:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698716.581624:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581638:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698716.581665:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698716.581677:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698716.581695:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698716.581697:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698716.581699:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698716.581702:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6617 00000100:00080000:2.0:1787698716.581704:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698716.581705:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581707:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581709:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698716.581711:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698716.581713:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581715:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581716:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698716.581718:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698716.581726:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698716.581729:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698716.581736:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698716.581743:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698716.581749:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698716.581751:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698716.581752:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698716.581754:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6617 00000100:00080000:2.0:1787698716.581756:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698716.581758:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581759:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581760:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698716.581761:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698716.581763:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581767:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581770:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698716.581774:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698716.581780:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698716.581782:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698716.581784:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698716.581797:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698716.581817:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.581827:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126373376:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698716.581832:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698716.581837:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698716.581839:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698716.581845:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6617 00000100:00080000:2.0:1787698716.581850:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698716.581853:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581855:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581857:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698716.581859:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698716.581866:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698716.581867:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698716.581877:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581887:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698716.581901:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698716.581905:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698716.581914:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698716.581919:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698716.581928:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698716.581936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581938:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698716.581946:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698716.581949:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698716.581951:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698716.581957:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6617 00000100:00000040:1.0:1787698716.581958:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698716.581964:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698716.581977:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581980:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581982:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698716.581984:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698716.581987:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698716.581991:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698716.581993:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698716.581995:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698716.581999:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698716.582001:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698716/real 1787698716] req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698716.582004:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698716.582018:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698716.582028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698716.582040:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698716.582066:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6627) 00000100:00080000:1.0:1787698716.582076:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698716.582081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698716.582089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582104:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126373376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698716.582113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698716.582154:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582159:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698716.582168:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698716.582173:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582180:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698716.582183:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698716/real 1787698716] req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582190:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582195:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698716.582200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698716.582201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698716.582203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698716.582211:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126373376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698716.582218:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698716.582225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126373504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698716.582229:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698716.582239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582245:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698716.582252:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582256:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698716.582261:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698716.582265:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582270:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698716.582273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698716/real 1787698716] req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582279:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582284:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698716.582289:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698716.582290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698716.582292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698716.582301:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698716.582312:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582316:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698716.582321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698716.582324:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582329:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698716.582332:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698716/real 1787698716] req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582342:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698716.582346:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698716.582348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698716.582349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698716.582357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582362:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698716.582368:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582371:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698716.582376:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698716.582379:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698741 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582384:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698716.582387:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698716/real 1787698716] req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698716.582397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698716.582401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698716.582402:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698716.582404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698716.582408:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126373888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698716.582410:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126373632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698716.582461:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698716.582468:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126373760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698716.582474:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698716.582477:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126373888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698741 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698716.582481:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698717.093577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698717.093577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698717.093577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698717.221612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698718.117499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698718.117500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698718.117808:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698718.245581:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698718.501507:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301291887, at 4301291904 00000100:00000040:3.0:1787698718.501508:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301291887, at 4301291904 00000100:00100000:0.0:1787698719.141544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698719.141544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698719.141555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698719.269602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698720.165504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698720.165504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698720.165504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698720.293575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698721.189524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698721.189524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698721.189524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698721.317599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698721.701497:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698721.701507:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698721.701511:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698721.701521:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6622 00000100:00080000:2.0:1787698721.701531:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698721.701535:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698721.701539:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701542:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698721.701546:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698721.701549:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698721.701552:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701555:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698721.701557:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701570:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698721.701594:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698721.701607:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698721.701638:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698721.701642:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698721.701644:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698721.701648:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6622 00000100:00080000:2.0:1787698721.701651:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698721.701654:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698721.701656:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701659:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698721.701661:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698721.701665:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698721.701668:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701670:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698721.701674:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698721.701682:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698721.701687:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698721.701690:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698721.701699:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698721.701726:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698721.701730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126374016:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698721.701744:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698721.701747:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698721.701749:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698721.701754:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6622 00000100:00080000:2.0:1787698721.701757:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698721.701760:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698721.701765:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698721.701766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701772:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698721.701777:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698721.701784:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698721.701788:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698721.701791:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701795:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698721.701798:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698721.701803:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698721.701808:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698721.701817:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698721.701833:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698721.701841:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698721.701845:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698721.701846:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698721.701851:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6622 00000100:00080000:2.0:1787698721.701855:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698721.701859:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698721.701864:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698721.701865:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698721.701868:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698721.701873:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698721.701879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698721.701885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698721.701886:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701891:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698721.701896:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698721.701902:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698721.701908:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698721.701908:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698721.701921:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698721.701923:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698721.701934:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698721.701937:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787698721.701948:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698721.701948:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698721.701957:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698721.701962:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698721.701967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374272:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698721.701971:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6622 00000100:00080000:2.0:1787698721.701978:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698721.701981:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698721.701985:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698721.701987:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.701992:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698721.701997:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698721.702004:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698721.702005:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698721.702010:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698721.702013:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698721.702016:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698721.702017:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000008:00100000:2.0:1787698721.702021:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698721.702041:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698721.702041:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698721.702054:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698721.702056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698721.702072:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698721.702077:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698721.702088:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6632) 00000100:00000040:1.0:1787698721.702098:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698721.702111:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787698721.702119:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698721.702127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698721.702137:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698721.702144:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698721.702152:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698722.213479:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698722.213520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698722.213520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698722.341585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698723.237534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698723.237534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698723.237534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698723.365559:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698723.621541:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301297007, at 4301297024 00000100:00000040:3.0:1787698723.621541:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301297007, at 4301297024 00000100:00100000:1.0:1787698724.261568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698724.261573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698724.261596:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698724.389625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698724.773755:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698724.773807:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698724.773826:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698724.773885:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698724.773907:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698724.773916:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698724.773928:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787698724.773939:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698721/real 1787698724] req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698724.773940:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698724.773968:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698724.773988:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698724.774000:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787698724.774005:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698724.774014:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698746 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698724.774026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698724.774052:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698724.774059:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698724.774064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774073:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374144:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698724.774081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698721/real 1787698724] req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774106:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698724.774117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698724.774133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698724.774137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698724.774141:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774149:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374272:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698724.774155:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698721/real 1787698724] req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774168:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698724.774188:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698724.774195:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698724.774199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698724.774202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774209:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126374016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698724.774215:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126374144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698724.774230:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698724.774246:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126374272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698724.774258:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698724.774263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126374016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698724.774269:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698724.774275:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698721/real 1787698724] req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774287:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698724.774295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698724.774302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698724.774305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698724.774309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374528:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698724.774322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698721/real 1787698724] req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698724.774341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698724.774353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698724.774356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698724.774360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698724.774369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698724.774374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126374528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698724.774382:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698724.774387:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126374400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698746 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698724.774395:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698725.285531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698725.285531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698725.285531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698725.413562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698726.309550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698726.309550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698726.309566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698726.437633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698726.821550:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698726.821558:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698726.821565:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698726.821573:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6627 00000100:00080000:0.0:1787698726.821582:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698726.821584:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821587:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821590:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698726.821594:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698726.821597:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821598:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821600:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698726.821603:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821612:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698726.821634:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698726.821644:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698726.821661:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698726.821663:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698726.821664:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698726.821668:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6627 00000100:00080000:0.0:1787698726.821671:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698726.821675:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821677:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821679:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698726.821682:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698726.821685:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821689:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821691:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698726.821694:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698726.821703:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698726.821709:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698726.821710:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698726.821724:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698726.821730:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126374656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698726.821742:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698726.821752:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698726.821755:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698726.821757:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698726.821761:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6627 00000100:00080000:0.0:1787698726.821766:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698726.821767:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698726.821772:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821779:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821783:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698726.821785:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698726.821787:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821790:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821795:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698726.821801:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698726.821802:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787698726.821810:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698726.821817:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698726.821826:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698726.821835:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698726.821844:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698726.821849:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1787698726.821850:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698726.821853:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698726.821860:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6627 00000100:00080000:0.0:1787698726.821864:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698726.821869:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821873:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821876:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:1.0:1787698726.821876:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698726.821880:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698726.821886:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821893:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698726.821899:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698726.821905:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698726.821912:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698726.821917:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698726.821927:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698726.821962:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698726.821970:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698726.821972:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698726/real 1787698726] req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698726.821974:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698726.821978:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698726.821985:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6627 00000100:00080000:0.0:1787698726.821991:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698726.821994:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.821998:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698726.822002:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698726.822003:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698726.822007:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698726.822014:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698726.822021:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698726.822023:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698726.822024:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698726.822030:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698726.822040:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698726.822042:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698726.822049:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698726.822050:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698726.822057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698726.822063:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698726.822077:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126374656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698726.822084:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822091:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126374656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698726.822100:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6637) 00000020:00000040:1.0:1787698726.822106:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698726.822126:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698726.822149:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822157:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698726.822170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698726.822178:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822194:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698726.822199:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698726/real 1787698726] req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698726.822226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698726.822229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698726.822233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822241:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698726.822247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822255:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698726.822267:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822275:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698726.822285:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698726.822292:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822301:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698726.822308:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698726/real 1787698726] req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822319:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698726.822337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698726.822341:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698726.822346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822354:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126374912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698726.822360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698726.822382:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822390:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698726.822402:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698726.822409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822447:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698726.822454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698726/real 1787698726] req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698726.822489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698726.822492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698726.822496:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698726.822509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822517:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698726.822528:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822535:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698726.822545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698726.822551:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698751 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822560:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698726.822565:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698726/real 1787698726] req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822575:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698726.822583:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698726.822590:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698726.822593:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698726.822597:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698726.822604:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698726.822610:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126374784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698726.822618:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698726.822623:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126374912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698726.822629:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698726.822634:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126375040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698726.822640:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698726.822644:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126375168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698751 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698726.822651:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698727.333520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698727.333520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698727.333524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698727.461622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698728.357511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698728.357511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698728.357511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698728.485587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698728.741524:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301302127, at 4301302144 00000100:00000040:3.0:1787698728.741524:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301302127, at 4301302144 00000100:00100000:0.0:1787698729.381514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698729.381514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698729.381514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698729.509583:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698730.405503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698730.405516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698730.405542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698730.533589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698731.429547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698731.429547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698731.429555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698731.557598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698731.941597:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698731.941611:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698731.941616:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698731.941629:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6632 00000100:00080000:2.0:1787698731.941644:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698731.941647:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941652:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941655:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698731.941668:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698731.941672:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941675:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941678:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698731.941681:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941698:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698731.941730:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698731.941746:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698731.941770:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698731.941773:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698731.941775:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698731.941779:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6632 00000100:00080000:2.0:1787698731.941782:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698731.941785:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941787:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941790:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698731.941793:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698731.941796:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941799:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941802:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698731.941804:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698731.941815:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698731.941819:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698731.941822:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698731.941838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126375296:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698731.941840:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698731.941863:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698731.941873:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698731.941876:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698731.941878:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698731.941881:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6632 00000100:00080000:2.0:1787698731.941885:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698731.941888:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941891:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941893:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698731.941896:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698731.941899:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941902:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941904:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787698731.941907:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698731.941909:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698731.941916:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698731.941920:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698731.941926:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698731.941933:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698731.941943:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698731.941951:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698731.941955:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698731.941956:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698731.941961:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6632 00000100:00080000:2.0:1787698731.941965:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698731.941967:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698731.941971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941973:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941977:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698731.941979:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698731.941982:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.941986:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.941988:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698731.941992:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698731.941994:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698731.941998:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698731.942006:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698731.942016:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698731.942025:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698731/real 1787698731] req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698731.942034:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698731.942041:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698731.942045:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698731.942052:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6632 00000100:00080000:2.0:1787698731.942055:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698731.942058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698731.942059:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.942065:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.942070:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698731.942073:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698731.942076:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698731.942080:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698731.942088:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698731.942091:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698731.942092:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698731.942095:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:1.0:1787698731.942099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:2.0:1787698731.942101:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698731.942103:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698731.942107:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698731.942114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126375296:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698731.942120:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698731.942124:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126375296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698731.942144:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698731.942144:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698731.942158:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6643) 00000100:00000040:1.0:1787698731.942162:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698731.942182:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942186:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698731.942194:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698731.942198:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942205:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698731.942209:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698731/real 1787698731] req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942215:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942221:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698731.942226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698731.942229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698731.942232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942245:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698731.942251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942259:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698731.942271:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942279:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698731.942289:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698731.942294:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942299:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698731.942304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698731/real 1787698731] req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698731.942321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698731.942322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698731.942324:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942329:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698731.942332:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698731.942346:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942350:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698731.942355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698731.942359:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942364:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698731.942367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698731/real 1787698731] req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698731.942383:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698731.942385:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698731.942386:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698731.942393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698731.942405:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942409:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698731.942414:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698731.942417:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698756 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942445:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698731.942449:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698731/real 1787698731] req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942456:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698731.942461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698731.942465:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698731.942467:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698731.942469:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698731.942472:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126375808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698731.942476:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126375424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698731.942481:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698731.942486:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126375552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698731.942490:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698731.942493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126375680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698731.942497:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698731.942500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126375808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698756 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698731.942504:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698732.453488:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698732.453531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698732.453784:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698732.581607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698733.477519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698733.477522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698733.477534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698733.605584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698733.861531:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301307247, at 4301307264 00000100:00000040:3.0:1787698733.861531:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301307247, at 4301307264 00000100:00100000:1.0:1787698734.501545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698734.501545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698734.501545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698734.629602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698735.525554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698735.525554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698735.525564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698735.653528:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698736.549562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698736.549562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698736.549563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698736.677645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698737.061547:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698737.061561:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698737.061565:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698737.061576:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6638 00000100:00080000:2.0:1787698737.061588:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698737.061591:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061595:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061597:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698737.061603:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698737.061605:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061606:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061609:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698737.061611:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061625:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698737.061653:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698737.061665:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.061689:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698737.061692:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698737.061694:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698737.061698:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6638 00000100:00080000:2.0:1787698737.061699:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698737.061701:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061703:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061704:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698737.061706:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698737.061708:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061710:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061712:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698737.061713:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698737.061723:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698737.061726:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698737.061736:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698737.061742:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.061748:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698737.061750:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698737.061751:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698737.061754:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6638 00000100:00080000:2.0:1787698737.061755:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698737.061757:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061759:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061760:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698737.061761:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698737.061774:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061782:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698737.061785:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698737.061785:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698737.061788:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698737.061796:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698737.061805:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698737.061819:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698737.061832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126375936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698737.061833:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.061860:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698737.061862:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698737.061863:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698737.061866:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6638 00000100:00080000:2.0:1787698737.061868:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698737.061870:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061872:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061874:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698737.061876:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698737.061878:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061879:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061881:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698737.061882:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698737.061885:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698737.061890:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698737.061898:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698737.061904:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698737.061922:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061933:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698737.061940:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698737.061943:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698737.061944:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698737.061946:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6638 00000100:00080000:2.0:1787698737.061951:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698737.061955:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061957:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061963:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000800:00000100:1.0:1787698737.061966:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061968:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698737.061973:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698737.061976:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698737.061979:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698737.061983:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698737.061988:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698737.061989:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698737.062004:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698737.062023:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698737.062035:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698737.062038:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698737.062061:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6648) 00000100:00100000:1.0:1787698737.062080:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698737.062101:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698737.062105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698737.062109:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062116:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126375936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698737.062122:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698737.062145:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062151:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698737.062163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698737.062170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062187:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698737.062193:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698737.062219:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698737.062222:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698737.062224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062229:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698737.062233:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126375936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698737.062244:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698737.062253:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126376064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698737.062260:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698737.062274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698737.062292:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062297:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698737.062305:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698737.062310:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062316:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698737.062319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062331:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698737.062337:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698737.062340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698737.062343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698737.062356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062365:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698737.062377:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062383:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698737.062398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698737.062403:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062413:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698737.062416:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062452:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698737.062462:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698737.062464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698737.062466:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062470:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698737.062474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062479:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698737.062486:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062490:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698737.062496:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698737.062500:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698762 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062505:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698737.062508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698737/real 1787698737] req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698737.062519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698737.062522:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698737.062524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698737.062526:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698737.062530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698737.062533:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126376192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698737.062538:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698737.062541:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126376320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698737.062545:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698737.062548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126376448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698762 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698737.062552:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698737.573508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698737.573508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698737.573547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698737.701603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698738.597556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698738.597557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698738.597557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698738.725596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698738.981533:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301312367, at 4301312384 00000100:00000040:2.0:1787698738.981534:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301312367, at 4301312384 00000100:00100000:1.0:1787698739.621519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698739.621519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698739.621547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698739.749597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698740.645536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698740.645536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698740.645537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698740.773606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698741.669483:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698741.669483:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698741.669503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698741.797538:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698742.181585:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698742.181595:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698742.181600:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698742.181610:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6643 00000100:00080000:1.0:1787698742.181622:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698742.181625:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.181628:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181631:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698742.181636:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698742.181638:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.181641:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181643:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698742.181645:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181660:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698742.181705:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698742.181722:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698742.181749:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698742.181753:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698742.181755:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698742.181760:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6643 00000100:00080000:1.0:1787698742.181764:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698742.181769:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.181771:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181774:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698742.181777:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698742.181781:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.181785:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181788:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698742.181791:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698742.181802:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698742.181811:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698742.181812:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698742.181833:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698742.181838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126376576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698742.181862:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698742.181869:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698742.181871:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698742.181872:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698742.181876:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6643 00000100:00080000:1.0:1787698742.181879:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698742.181881:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698742.181887:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698742.181895:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181898:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:2.0:1787698742.181900:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181906:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698742.181911:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.181913:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181915:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698742.181918:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698742.181922:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698742.181926:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698742.181935:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698742.181942:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698742.181954:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698742.181960:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698742.181962:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:2.0:1787698742.181965:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698742.181966:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698742.181978:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6643 00000100:00080000:1.0:1787698742.181985:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698742.181989:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.181993:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.181994:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698742.181996:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698742.181999:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.182000:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.182002:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698742.182005:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698742.182009:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698742.182013:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698742.182021:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698742/real 1787698742] req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698742.182022:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698742.182082:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698742.182088:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698742.182090:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698742.182095:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698742.182098:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698742.182102:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6643 00000100:00080000:1.0:1787698742.182115:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698742.182121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698742.182127:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698742.182139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:1.0:1787698742.182140:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698742.182146:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787698742.182148:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698742.182154:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698742.182154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698742.182162:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698742.182169:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698742.182174:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698742.182183:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698742.182196:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698742.182197:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126376576:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698742.182204:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698742.182210:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126376576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698742.182217:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787698742.182224:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698742.182233:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698742.182241:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6653) 00000100:00000040:2.0:1787698742.182248:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182257:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698742.182272:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182279:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698742.182293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698742.182301:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182315:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698742.182319:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698742/real 1787698742] req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182326:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182332:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698742.182336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698742.182338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698742.182340:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698742.182349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182354:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698742.182362:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182366:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698742.182373:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698742.182377:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182382:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698742.182386:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698742/real 1787698742] req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182392:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698742.182402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698742.182404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698742.182406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182410:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698742.182413:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182436:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698742.182444:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182449:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698742.182454:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698742.182458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182464:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698742.182467:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698742/real 1787698742] req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182478:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698742.182487:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698742.182488:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698742.182490:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182494:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126376960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698742.182498:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182503:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698742.182508:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182512:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698742.182517:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698742.182520:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698767 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182525:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698742.182528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698742/real 1787698742] req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698742.182539:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698742.182543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698742.182544:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698742.182546:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698742.182550:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698742.182553:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126376704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698742.182559:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698742.182564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126376832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698742.182570:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698742.182575:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126376960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698742.182581:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698742.182585:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126377088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698767 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698742.182593:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698742.693545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698742.693545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698742.693545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698742.821527:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698743.717549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698743.717550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698743.717551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698743.845620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698744.101580:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301317487, at 4301317504 00000100:00000040:3.0:1787698744.101597:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301317487, at 4301317504 00000100:00100000:1.0:1787698744.741539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698744.741540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698744.741540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698744.869597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698745.765515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698745.765515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698745.765515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698745.893652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698746.789529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698746.789535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698746.789595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698746.917594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698747.301621:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698747.301636:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698747.301641:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698747.301655:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6648 00000100:00080000:0.0:1787698747.301668:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698747.301671:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.301674:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.301678:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698747.301683:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698747.301685:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.301687:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.301690:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698747.301693:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698747.301715:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698747.301749:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698747.301763:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698747.301784:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698747.301788:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698747.301790:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698747.301794:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6648 00000100:00080000:0.0:1787698747.301799:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698747.301802:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.301805:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.301808:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698747.301812:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698747.301816:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.301820:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.301823:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698747.301827:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698747.301840:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698747.301847:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698747.301863:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698747.301864:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698747.301944:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698747.301967:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698747.301967:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126377216:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698747.301982:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698747.301985:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698747.301989:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6648 00000100:00080000:0.0:1787698747.301992:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698747.301995:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.301998:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.302001:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698747.302004:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698747.302007:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.302009:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.302011:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698747.302013:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698747.302018:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698747.302018:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698747.302028:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698747.302034:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698747.302041:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698747.302052:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698747.302068:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698747.302073:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787698747.302074:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698747.302076:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698747.302082:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6648 00000100:00080000:0.0:1787698747.302086:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698747.302089:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.302091:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.302095:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698747.302098:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698747.302098:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698747.302106:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.302116:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.302122:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698747.302134:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698747.302142:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698747.302146:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698747.302154:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698747.302163:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698747.302171:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698747.302175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698747/real 1787698747] req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698747.302177:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698747.302181:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698747.302191:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6648 00000100:00080000:0.0:1787698747.302200:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698747.302206:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.302210:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.302214:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:2.0:1787698747.302216:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698747.302220:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698747.302229:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698747.302242:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698747.302245:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787698747.302249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698747.302254:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698747.302268:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698747.302271:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698747.302276:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698747.302281:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698747.302286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126377216:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698747.302301:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698747.302314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126377216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698747.302327:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698747.302341:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698747.302344:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6658) 00000100:00000040:2.0:1787698747.302361:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698747.302389:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302395:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698747.302405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698747.302411:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302458:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698747.302464:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698747/real 1787698747] req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302473:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302479:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698747.302485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698747.302487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698747.302490:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698747.302499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698747.302514:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302520:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698747.302527:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698747.302531:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302538:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698747.302541:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698747/real 1787698747] req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302549:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302554:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698747.302560:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698747.302562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698747.302564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698747.302572:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302578:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698747.302586:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302609:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698747.302633:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698747.302642:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302652:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698747.302658:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698747/real 1787698747] req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302666:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302672:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698747.302677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698747.302679:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698747.302681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302686:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698747.302690:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302695:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698747.302705:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302710:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698747.302719:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698747.302723:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698772 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302729:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698747.302732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698747/real 1787698747] req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698747.302745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698747.302750:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698747.302751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698747.302754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698747.302758:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698747.302761:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126377344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698747.302766:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698747.302770:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126377472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698747.302775:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698747.302778:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126377600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698747.302783:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698747.302786:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126377728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698772 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698747.302791:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698747.813543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698747.813544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698747.813559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698747.941653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698748.837527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698748.837530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698748.837550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698748.965613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698749.221521:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301322607, at 4301322624 00000100:00000040:2.0:1787698749.221573:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301322607, at 4301322624 00000100:00100000:0.0:1787698749.861562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698749.861562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698749.861568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698749.989669:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698750.885565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698750.885565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698750.885584:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698751.013610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698751.909508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698751.909509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698751.909558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698752.037666:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698752.421524:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698752.421535:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698752.421541:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698752.421555:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6653 00000100:00080000:0.0:1787698752.421571:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698752.421575:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421580:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421584:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698752.421590:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698752.421594:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421596:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421600:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698752.421603:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421620:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698752.421654:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698752.421671:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698752.421702:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698752.421705:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698752.421707:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698752.421710:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6653 00000100:00080000:0.0:1787698752.421715:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698752.421717:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421719:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421721:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698752.421723:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698752.421725:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421727:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421729:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698752.421731:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698752.421740:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698752.421744:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698752.421753:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698752.421761:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698752.421768:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698752.421771:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698752.421772:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698752.421775:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6653 00000100:00080000:0.0:1787698752.421778:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698752.421783:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421785:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698752.421786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698752.421787:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698752.421790:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698752.421795:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421799:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421802:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698752.421804:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698752.421813:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698752.421817:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698752.421822:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126377856:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698752.421826:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698752.421835:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698752.421841:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698752.421843:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698752.421845:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698752.421847:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6653 00000100:00080000:0.0:1787698752.421850:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698752.421852:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421854:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421856:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698752.421858:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698752.421861:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698752.421864:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421870:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421875:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698752.421880:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698752.421881:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698752.421888:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698752.421893:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698752.421901:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698752.421912:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698752.421918:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698752.421920:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698752.421921:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698752.421924:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6653 00000100:00080000:0.0:1787698752.421930:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698752.421930:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421937:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421939:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421940:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698752.421942:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698752.421944:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698752.421946:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698752.421948:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698752.421950:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698752.421954:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698752.421959:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698752.421959:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698752.421967:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698752.421983:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698752.421991:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6663) 00000100:00100000:2.0:1787698752.422011:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698752/real 1787698752] req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698752.422040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698752.422046:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698752.422051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422061:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126377856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698752.422068:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422079:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698752.422096:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422102:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698752.422112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698752.422118:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422126:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698752.422131:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698752/real 1787698752] req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422139:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698752.422151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698752.422153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698752.422155:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422161:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126377984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698752.422166:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126377856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698752.422174:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698752.422182:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126377984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698752.422186:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698752.422195:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422202:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698752.422210:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422215:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698752.422221:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698752.422225:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422232:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698752.422235:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698752/real 1787698752] req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698752.422255:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698752.422258:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698752.422260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698752.422268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422273:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698752.422280:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422285:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698752.422290:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698752.422294:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422301:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698752.422304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698752/real 1787698752] req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422311:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422317:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698752.422321:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698752.422323:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698752.422326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698752.422334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698752.422346:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422350:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698752.422355:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698752.422359:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698777 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422364:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698752.422367:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698752/real 1787698752] req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422375:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698752.422381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698752.422385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698752.422386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698752.422389:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698752.422394:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698752.422398:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126378112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698752.422402:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698752.422406:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126378240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698752.422410:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698752.422413:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126378368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698777 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698752.422533:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698752.933518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698752.933518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698752.933541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698753.061654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698753.957527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698753.957527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698753.957548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698754.085666:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698754.341540:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301327727, at 4301327744 00000100:00000040:3.0:1787698754.341540:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301327727, at 4301327744 00000100:00100000:1.0:1787698754.981565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698754.981578:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698754.981615:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698755.109645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698756.005475:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698756.005503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698756.005553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698756.133647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698757.029513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698757.029513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698757.029513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698757.157677:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698757.541617:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698757.541629:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698757.541633:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698757.541644:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6658 00000100:00080000:1.0:1787698757.541658:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698757.541660:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.541664:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.541668:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698757.541675:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698757.541678:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.541682:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.541687:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698757.541691:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698757.541711:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698757.541747:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698757.541767:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698757.541795:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698757.541800:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698757.541803:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698757.541808:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6658 00000100:00080000:1.0:1787698757.541814:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698757.541818:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.541821:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.541824:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698757.541828:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698757.541832:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.541835:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.541837:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698757.541841:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698757.541855:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698757.541863:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698757.541875:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698757.541892:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698757.541893:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698757.541966:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698757.541972:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126378496:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698757.541973:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698757.541976:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698757.541992:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6658 00000100:00080000:1.0:1787698757.541998:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698757.542003:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.542007:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.542009:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698757.542012:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698757.542016:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.542021:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.542028:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787698757.542029:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698757.542037:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698757.542053:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698757.542063:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698757.542063:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698757.542084:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698757.542093:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698757.542101:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698757.542103:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698757.542105:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698757.542109:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6658 00000100:00080000:1.0:1787698757.542116:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698757.542116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698757.542128:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.542133:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.542136:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698757.542140:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698757.542147:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698757.542148:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698757.542157:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.542166:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698757.542175:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698757.542185:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698757.542191:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698757.542202:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698757.542210:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698757.542217:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698757.542223:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698757/real 1787698757] req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698757.542224:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698757.542231:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698757.542241:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6658 00000100:00080000:1.0:1787698757.542250:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698757.542266:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.542274:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698757.542278:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698757.542280:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698757.542284:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698757.542288:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698757.542292:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698757.542294:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698757.542307:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698757.542319:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698757.542328:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698757.542328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698757.542348:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698757.542350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698757.542368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698757.542381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698757.542385:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698757.542488:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6668) 00000100:00100000:2.0:1787698757.542495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126378496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698757.542508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126378496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698757.542525:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698757.542545:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542556:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698757.542573:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542582:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698757.542598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698757.542606:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542618:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698757.542624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698757/real 1787698757] req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542639:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542650:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698757.542658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698757.542662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698757.542667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542677:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698757.542684:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542694:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698757.542708:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542717:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698757.542729:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698757.542736:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542747:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698757.542753:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698757/real 1787698757] req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542774:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542785:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698757.542794:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698757.542799:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698757.542803:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542812:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698757.542819:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698757.542844:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542853:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698757.542865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698757.542872:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542882:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698757.542888:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698757/real 1787698757] req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542908:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542918:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698757.542926:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698757.542929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698757.542932:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126378880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698757.542942:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542948:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698757.542958:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542963:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698757.542971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698757.542976:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698782 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542983:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698757.542986:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698757/real 1787698757] req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.542993:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698757.542999:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698757.543004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698757.543006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698757.543008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698757.543013:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698757.543016:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126378624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698757.543021:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698757.543025:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126378752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698757.543029:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698757.543032:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126378880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698757.543037:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698757.543040:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126379008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698782 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698757.543044:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698758.053555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698758.053555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698758.053558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698758.181582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698759.077529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698759.077529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698759.077570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698759.205624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698759.461493:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301332847, at 4301332864 00000100:00000040:3.0:1787698759.461549:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301332847, at 4301332864 00000100:00100000:1.0:1787698760.101579:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698760.101579:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698760.101582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698760.229588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698761.125554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698761.125554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698761.125595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698761.253783:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698762.149539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698762.149539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698762.149550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698762.277599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698762.661544:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698762.661551:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698762.661555:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698762.661563:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6663 00000100:00080000:1.0:1787698762.661572:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698762.661575:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661579:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661582:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698762.661586:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698762.661588:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661591:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661593:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698762.661595:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661608:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698762.661633:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698762.661650:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698762.661684:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698762.661687:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698762.661689:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698762.661693:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6663 00000100:00080000:1.0:1787698762.661696:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698762.661698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661700:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661703:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698762.661706:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698762.661708:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661712:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698762.661714:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698762.661723:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698762.661727:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698762.661737:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698762.661749:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698762.661766:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698762.661768:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698762.661771:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698762.661775:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698762.661790:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6663 00000100:00100000:2.0:1787698762.661807:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126379136:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698762.661809:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698762.661817:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661821:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661824:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698762.661827:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698762.661833:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661835:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661837:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698762.661839:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698762.661845:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698762.661848:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698762.661851:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698762.661869:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698762.661878:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698762.661894:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698762.661905:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698762.661910:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787698762.661917:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698762.661922:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698762.661929:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6663 00000100:00080000:1.0:1787698762.661936:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698762.661941:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698762.661942:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661958:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661971:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698762.661976:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698762.661979:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.661985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.661987:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698762.661990:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698762.661994:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698762.661998:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698762.662013:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698762/real 1787698762] req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698762.662014:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698762.662071:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698762.662081:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698762.662081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698762.662100:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698762.662107:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698762.662111:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6663 00000100:00080000:1.0:1787698762.662115:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698762.662117:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.662124:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.662125:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698762.662127:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698762.662130:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698762.662132:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698762.662134:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698762.662140:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698762.662146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698762.662150:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698762.662164:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698762.662170:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787698762.662176:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787698762.662177:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698762.662194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698762.662216:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662225:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126379136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698762.662236:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6673) 00000100:00000040:2.0:1787698762.662244:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126379136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698762.662261:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698762.662281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698762.662306:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662311:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698762.662321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698762.662326:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662359:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698762.662365:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698762/real 1787698762] req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698762.662385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698762.662387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698762.662390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698762.662406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662416:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698762.662472:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662478:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698762.662487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698762.662493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662500:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698762.662504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698762/real 1787698762] req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662512:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698762.662525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698762.662526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698762.662529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698762.662538:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662544:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698762.662553:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662558:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698762.662564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698762.662568:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662584:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698762.662588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698762/real 1787698762] req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662602:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698762.662607:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698762.662609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698762.662611:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662616:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698762.662620:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662626:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698762.662634:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662643:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698762.662650:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698762.662655:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698787 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662661:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698762.662664:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698762/real 1787698762] req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662671:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698762.662677:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698762.662682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698762.662684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698762.662686:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698762.662691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698762.662697:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126379264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698762.662705:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698762.662712:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126379392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698762.662719:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698762.662725:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126379520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698762.662732:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698762.662739:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126379648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698787 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698762.662747:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698763.173512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698763.173512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698763.173512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698763.301591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698764.197571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698764.197571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698764.197572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698764.325637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698764.581518:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301337967, at 4301337984 00000100:00000040:3.0:1787698764.581518:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301337967, at 4301337984 00000100:00100000:0.0:1787698765.221589:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698765.221589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698765.221589:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698765.349624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698766.245559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698766.245559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698766.245559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698766.373649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698767.269534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698767.269538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698767.269640:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698767.397662:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698767.781555:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698767.781569:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698767.781575:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698767.781589:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6668 00000100:00080000:2.0:1787698767.781604:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698767.781608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.781612:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.781617:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698767.781623:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698767.781626:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.781629:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.781633:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698767.781636:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698767.781656:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698767.781702:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698767.781719:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698767.781750:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698767.781755:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698767.781757:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698767.781763:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6668 00000100:00080000:2.0:1787698767.781770:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698767.781774:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.781777:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.781779:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698767.781782:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698767.781785:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.781787:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.781789:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698767.781793:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698767.781804:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698767.781807:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698767.781815:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698767.781835:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698767.781838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126379776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698767.781866:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698767.781874:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698767.781876:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698767.781878:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698767.781882:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6668 00000100:00080000:2.0:1787698767.781885:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698767.781889:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698767.781894:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.781909:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.781920:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698767.781925:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698767.781926:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698767.781940:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.781944:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.781946:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698767.781949:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698767.781953:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698767.781957:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698767.781964:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698767.781974:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698767.781975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698767.781988:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698767.781990:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698767.781993:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698767.782002:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698767.782002:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6668 00000100:00080000:2.0:1787698767.782011:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698767.782023:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.782036:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.782042:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698767.782047:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698767.782052:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.782056:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.782058:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698767.782062:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698767.782069:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698767.782075:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698767.782081:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698767/real 1787698767] req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698767.782091:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698767.782112:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698767.782121:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782129:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698767.782169:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698767.782172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698767.782175:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698767.782179:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:2.0:1787698767.782181:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698767.782186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698767.782193:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6668 00000100:00100000:1.0:1787698767.782206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126379776:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698767.782207:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698767.782221:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698767.782222:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126379776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698767.782253:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.782266:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698767.782272:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000020:00000040:1.0:1787698767.782273:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698767.782280:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698767.782285:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698767.782288:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698767.782291:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698767.782298:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698767.782303:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698767.782307:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698767.782332:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379904:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787698767.782332:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698767.782356:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782365:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698767.782379:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6678) 00000100:00000040:1.0:1787698767.782386:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698767.782411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698767.782448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782467:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698767.782473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698767/real 1787698767] req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698767.782493:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698767.782495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698767.782498:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126379904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698767.782508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782514:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698767.782523:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782528:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698767.782537:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698767.782541:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782548:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698767.782552:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698767/real 1787698767] req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782559:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782565:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698767.782571:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698767.782573:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698767.782575:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782579:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698767.782583:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782590:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698767.782598:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782603:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698767.782608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698767.782612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782625:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698767.782630:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698767/real 1787698767] req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782648:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698767.782652:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698767.782654:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698767.782657:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698767.782665:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126379904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698767.782670:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698767.782675:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126380032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698767.782680:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698767.782683:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126380160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698767.782687:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698767.782693:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782699:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698767.782708:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782712:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698767.782719:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698767.782723:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698792 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782729:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698767.782732:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698767/real 1787698767] req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782740:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698767.782745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698767.782749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698767.782751:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698767.782754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698767.782761:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698767.782767:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126380288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698792 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698767.782775:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698768.293552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698768.293552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698768.293576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698768.421587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698769.317509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698769.317509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698769.317509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698769.445622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698769.701545:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301343087, at 4301343104 00000100:00000040:3.0:1787698769.701545:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301343087, at 4301343104 00000100:00100000:0.0:1787698770.341546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698770.341546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698770.341546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698770.469554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698771.365504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698771.365504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698771.365553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698771.493685:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698772.389520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698772.389549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698772.389549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698772.517585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698772.901550:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698772.901565:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698772.901572:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698772.901586:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6673 00000100:00080000:2.0:1787698772.901601:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698772.901605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.901610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901614:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698772.901621:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698772.901625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.901629:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698772.901634:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901650:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698772.901688:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698772.901708:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698772.901737:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698772.901741:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698772.901743:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698772.901748:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6673 00000100:00080000:2.0:1787698772.901754:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698772.901759:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.901763:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901767:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698772.901770:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698772.901774:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.901778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901781:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698772.901785:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698772.901792:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698772.901800:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698772.901811:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698772.901818:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126380416:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698772.901831:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698772.901844:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698772.901856:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698772.901858:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698772.901867:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698772.901873:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698772.901880:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698772.901883:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6673 00000100:00080000:2.0:1787698772.901890:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698772.901893:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.901897:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901900:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698772.901902:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698772.901906:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.901909:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901912:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698772.901916:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698772.901922:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787698772.901927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698772.901930:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698772.901944:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698772.901950:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698772.901970:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698772.901981:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698772.901984:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698772.901986:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698772.901993:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6673 00000100:00100000:1.0:1787698772.901996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698772/real 1787698772] req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698772.902000:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698772.902009:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.902018:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.902022:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698772.902025:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698772.902028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698772.902033:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.902042:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.902048:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698772.902051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698772.902056:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698772.902068:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698772.902069:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:2.0:1787698772.902078:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698772.902087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:2.0:1787698772.902093:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698772.902097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698772.902119:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902121:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126380416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698772.902136:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126380416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698772.902139:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698772.902150:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698772.902154:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:1.0:1787698772.902179:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:2.0:1787698772.902187:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6673 00000100:00080000:2.0:1787698772.902193:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698772.902197:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698772.902203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698772.902205:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.902212:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698772.902218:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698772.902223:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698772.902228:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698772.902234:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698772.902237:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698772.902240:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698772.902245:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698772.902249:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698772.902260:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698772.902261:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698772.902278:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698772.902279:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698772.902290:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698772.902300:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902307:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698772.902318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698772/real 1787698772] req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698772.902322:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6684) 00000100:00100000:1.0:1787698772.902337:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902348:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698772.902357:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698772.902360:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698772.902364:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902373:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698772.902383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902393:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698772.902403:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902412:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698772.902447:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698772.902454:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902464:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698772.902471:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698772/real 1787698772] req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902485:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698772.902513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698772.902516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698772.902521:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902530:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698772.902536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902546:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698772.902559:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902568:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698772.902578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698772.902585:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902595:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698772.902601:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698772/real 1787698772] req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902619:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902629:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698772.902638:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698772.902641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698772.902645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698772.902659:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126380544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698772.902668:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698772.902675:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126380672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698772.902683:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698772.902689:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126380800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698772.902697:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698772.902704:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902713:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698772.902724:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902732:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698772.902742:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698772.902748:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698797 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902757:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698772.902763:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698772/real 1787698772] req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902775:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698772.902785:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698772.902793:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698772.902796:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698772.902802:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698772.902810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126380928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698772.902816:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126380928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698797 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698772.902824:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698773.413566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698773.413566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698773.413691:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698773.541667:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698774.437550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698774.437550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698774.437550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698774.565647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698774.821539:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301348207, at 4301348224 00000100:00000040:3.0:1787698774.821539:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301348207, at 4301348224 00000100:00100000:3.0:1787698775.461566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698775.461895:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698775.462038:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698775.589632:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698776.485511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698776.485511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698776.485513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698776.613613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698777.509549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698777.509549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698777.509552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698777.637602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698778.021567:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698778.021580:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698778.021586:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698778.021598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6679 00000100:00080000:0.0:1787698778.021610:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698778.021614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021624:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021628:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698778.021634:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698778.021637:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021640:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021643:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698778.021647:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021665:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698778.021698:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698778.021712:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.021742:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698778.021745:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698778.021746:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698778.021749:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6679 00000100:00080000:0.0:1787698778.021752:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698778.021754:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021755:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021757:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698778.021759:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698778.021762:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021763:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021765:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698778.021767:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698778.021776:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698778.021780:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698778.021788:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698778.021795:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.021805:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698778.021808:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698778.021811:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698778.021815:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6679 00000100:00080000:0.0:1787698778.021818:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698778.021821:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021825:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021827:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698778.021829:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698778.021833:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021837:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021843:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698778.021846:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698778.021847:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698778.021851:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698778.021857:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698778.021868:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698778.021878:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.021881:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126381056:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698778.021905:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698778.021908:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698778.021910:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698778.021915:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6679 00000100:00080000:0.0:1787698778.021917:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698778.021919:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021921:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021922:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698778.021925:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698778.021928:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021930:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.021932:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698778.021933:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698778.021938:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698778.021940:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698778.021944:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698778.021954:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698778.021964:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698778.021969:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698778.021976:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698778.021978:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698778.021979:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698778.021981:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6679 00000100:00080000:0.0:1787698778.021998:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698778.022000:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.022002:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.022003:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698778.022004:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698778.022006:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698778.022012:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698778.022017:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000800:00000100:1.0:1787698778.022018:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698778.022022:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698778.022027:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698778.022031:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698778.022038:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698778.022048:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022048:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698778.022100:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6689) 00000100:00100000:1.0:1787698778.022126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022155:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698778.022164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698778.022168:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698778.022172:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022180:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126381056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698778.022187:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022196:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698778.022211:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022219:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698778.022234:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698778.022241:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022251:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698778.022258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022277:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698778.022285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698778.022289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698778.022293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698778.022308:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126381056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698778.022318:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698778.022328:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126381184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698778.022335:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698778.022350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022360:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698778.022372:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022377:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698778.022387:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698778.022394:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022403:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698778.022409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022468:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022476:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698778.022484:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698778.022487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698778.022491:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022498:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698778.022504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022513:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698778.022527:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022534:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698778.022547:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698778.022554:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022564:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698778.022570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022587:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022595:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698778.022602:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698778.022606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698778.022609:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022617:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698778.022623:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022631:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698778.022644:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022651:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698778.022662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698778.022669:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698803 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022677:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698778.022682:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698778/real 1787698778] req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698778.022700:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698778.022708:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698778.022711:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698778.022715:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698778.022722:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698778.022727:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126381312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698778.022734:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698778.022740:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126381440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698778.022746:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698778.022751:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126381568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698803 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698778.022758:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698778.533521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698778.533534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698778.533535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698778.661611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698779.557551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698779.557551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698779.557551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698779.687108:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698779.941554:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301353327, at 4301353344 00000100:00000040:3.0:1787698779.941554:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301353327, at 4301353344 00000100:00100000:3.0:1787698780.581501:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698780.581568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698780.581594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698780.709619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698781.605518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698781.605540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698781.605540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698781.733532:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698782.629546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698782.629547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698782.629586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698782.757590:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698783.141535:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698783.141544:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698783.141548:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698783.141558:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6684 00000100:00080000:1.0:1787698783.141568:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698783.141571:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141575:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.141578:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698783.141583:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698783.141586:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141589:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.141592:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698783.141595:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698783.141607:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698783.141631:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698783.141646:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698783.141671:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698783.141675:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698783.141677:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698783.141681:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6684 00000100:00080000:1.0:1787698783.141684:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698783.141686:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141689:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.141691:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698783.141694:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698783.141698:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141702:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.141704:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698783.141707:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698783.141714:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698783.141716:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698783.141722:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698783.141732:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698783.141737:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126381696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698783.141750:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698783.141765:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698783.141767:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698783.141771:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698783.141775:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698783.141779:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698783.141782:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6684 00000100:00080000:1.0:1787698783.141788:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698783.141791:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141794:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.141797:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698783.141799:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698783.141802:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141805:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.141807:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000800:00000100:2.0:1787698783.141811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698783.141814:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698783.141820:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698783.141825:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698783.141826:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698783.141838:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698783.141849:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698783.141857:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698783.141862:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787698783.141862:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698783.141866:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698783.141874:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6684 00000100:00080000:1.0:1787698783.141880:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698783.141883:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141888:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698783.141888:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698783.141893:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698783.141896:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698783.141906:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698783.141912:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.141919:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787698783.141921:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787698783.141922:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698783.141928:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787698783.141931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:1.0:1787698783.141935:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698783.141938:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698783.141943:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698783.141952:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126381696:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698783.141957:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698783.141961:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126381696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698783.141994:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698783.142007:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142011:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381824:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698783.142037:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698783.142043:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698783.142046:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698783.142051:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6684 00000100:00000040:2.0:1787698783.142055:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698783.142056:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698783.142062:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.142069:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.142073:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698783.142074:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698783.142077:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698783.142082:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698783.142085:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698783.142087:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698783.142090:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698783.142095:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1787698783.142099:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698783.142102:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698783.142110:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698783.142114:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698783.142121:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787698783.142129:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142130:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142157:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698783.142159:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6694) 00000100:00000040:2.0:1787698783.142167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.142172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.142175:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698783.142178:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698783.142186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698783.142199:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142203:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698783.142209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698783.142212:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142218:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698783.142221:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.142236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.142238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698783.142240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142252:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126381952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698783.142257:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126381824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698783.142266:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698783.142272:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126381952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698783.142278:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698783.142285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142293:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698783.142304:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142310:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698783.142321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698783.142327:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142335:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698783.142340:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.142363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.142364:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698783.142367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142371:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698783.142374:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698783.142386:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142390:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698783.142396:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698783.142400:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698808 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142405:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698783.142408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698783/real 1787698783] req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698783.142442:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698783.142447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698783.142449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698783.142451:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698783.142456:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698783.142459:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126382080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698783.142463:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698783.142466:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126382208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698808 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698783.142470:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698783.653485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698783.653507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698783.654460:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698783.781617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698784.677488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698784.677513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698784.677523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698784.805571:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698785.061517:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301358447, at 4301358464 00000100:00000040:3.0:1787698785.061517:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301358447, at 4301358464 00000100:00100000:2.0:1787698785.701565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698785.701570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698785.701695:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698785.829602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698786.725518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698786.725518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698786.725519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698786.853627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698787.749548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698787.749548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698787.749548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698787.877642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698788.261560:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698788.261569:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698788.261573:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698788.261583:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6689 00000100:00080000:2.0:1787698788.261595:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698788.261597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261600:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261603:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698788.261608:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698788.261610:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261612:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261615:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698788.261617:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261631:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698788.261657:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698788.261670:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698788.261689:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698788.261692:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698788.261693:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698788.261696:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6689 00000100:00080000:2.0:1787698788.261699:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698788.261701:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261703:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261705:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698788.261707:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698788.261709:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261711:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261712:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698788.261714:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698788.261719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698788.261721:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698788.261728:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698788.261739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126382336:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698788.261740:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698788.261752:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698788.261761:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698788.261763:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698788.261765:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698788.261767:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6689 00000100:00080000:2.0:1787698788.261769:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698788.261770:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261772:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261773:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698788.261775:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698788.261777:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698788.261780:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261785:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261788:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698788.261792:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698788.261796:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:2.0:1787698788.261796:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698788.261801:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698788.261809:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698788.261815:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698788.261820:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698788.261823:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698788.261824:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698788.261827:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6689 00000100:00080000:2.0:1787698788.261829:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698788.261832:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261834:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261835:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698788.261837:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698788.261839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261840:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261842:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698788.261843:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698788.261846:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698788.261850:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698788.261856:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698788.261862:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698788.261867:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698788.261869:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698788.261871:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698788.261874:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698788.261877:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6689 00000100:00080000:2.0:1787698788.261883:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698788.261888:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698788.261907:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698788.261907:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261910:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698788.261912:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698788.261917:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261920:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698788.261921:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698788.261923:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698788.261927:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698788.261931:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698788.261932:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698788.261941:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698788.261949:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698788.261954:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698788.261964:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6699) 00000100:00000040:1.0:1787698788.261970:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787698788.261980:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698788.261991:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698788.262006:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698788.262012:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698788.262023:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787698788.262027:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698788.262033:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698788.262042:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698788.262046:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698788.262053:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787698788.262062:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698788.262068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698788.262074:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698788.262079:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698788.262085:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698788.773515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698788.773515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698788.773515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698788.901646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698789.797547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698789.797547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698789.797556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698789.925645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698790.181535:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301363567, at 4301363584 00000100:00000040:3.0:1787698790.181536:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301363567, at 4301363584 00000100:00100000:3.0:1787698790.821480:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698790.821509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698790.821537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698790.949598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787698791.333749:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787698791.333795:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787698791.333812:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698791.333872:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698791.333895:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:1.0:1787698791.333902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698788/real 1787698791] req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698791.333909:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698791.333928:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698791.333942:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.333946:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698791.333957:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787698791.333969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698791.333970:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698791.334000:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1787698791.334005:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00080000:1.0:1787698791.334010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787698791.334018:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698813 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698791.334020:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334039:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126382336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698791.334047:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126382336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698791.334063:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698791.334077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698788/real 1787698791] req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334091:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698791.334101:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698791.334109:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698791.334113:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698791.334117:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334125:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382848:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698791.334132:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698788/real 1787698791] req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698791.334159:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698791.334167:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698791.334170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698791.334174:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334183:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382464:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698791.334190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698788/real 1787698791] req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698791.334212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698791.334220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698791.334223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698791.334227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382592:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698791.334241:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698788/real 1787698791] req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334253:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698791.334261:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698791.334268:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698791.334271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698791.334274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698791.334286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126382720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698791.334291:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126382848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698791.334299:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698791.334305:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126382464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698791.334312:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698791.334317:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126382592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698791.334325:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698791.334331:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126382720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698813 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698791.334337:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698791.845497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698791.845512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698791.845521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698791.973576:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698792.869467:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698792.869500:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698792.869532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698792.997569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698793.381575:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698793.381585:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698793.381589:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698793.381598:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6694 00000100:00080000:0.0:1787698793.381608:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698793.381611:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381615:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381618:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698793.381623:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698793.381626:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381629:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381632:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698793.381635:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381646:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698793.381679:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698793.381693:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698793.381720:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698793.381724:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698793.381726:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698793.381729:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6694 00000100:00080000:0.0:1787698793.381733:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698793.381736:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381739:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381741:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698793.381744:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698793.381747:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381749:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381751:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698793.381753:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698793.381768:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698793.381772:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698793.381782:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698793.381791:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698793.381799:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698793.381802:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698793.381804:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698793.381822:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6694 00000100:00080000:0.0:1787698793.381826:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698793.381829:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381831:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381833:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698793.381836:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698793.381838:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381840:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381843:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698793.381845:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698793.381850:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698793.381854:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698793.381861:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698793.381871:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698793.381878:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698793.381881:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698793.381883:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698793.381887:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6694 00000100:00080000:0.0:1787698793.381890:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698793.381892:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381895:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381897:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698793.381899:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698793.381901:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381904:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381906:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698793.381908:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698793.381912:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698793.381916:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698793.381923:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698793.381937:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698793.381944:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698793.381947:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698793.381949:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698793.381952:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6694 00000100:00080000:0.0:1787698793.381955:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698793.381957:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381962:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698793.381964:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698793.381967:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698793.381969:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698793.381971:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698793.381974:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698793.381978:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698793.381981:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698793.381988:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698793.381996:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698793.382005:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6704) 00000100:00000040:1.0:1787698793.382086:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382101:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126382976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698793.382126:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382135:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698793.382169:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698793.382183:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698793/real 1787698793] req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382230:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698793.382248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698793.382252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698793.382256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382264:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126382976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698793.382271:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698793.382300:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382307:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698793.382317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698793.382323:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382332:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698793.382339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698793/real 1787698793] req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382361:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698793.382368:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698793.382371:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698793.382375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698793.382386:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698793.382407:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382414:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698793.382459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698793.382469:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382477:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698793.382483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698793/real 1787698793] req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382506:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698793.382513:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698793.382516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698793.382519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698793.382531:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382538:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698793.382550:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382557:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698793.382568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698793.382574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382581:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698793.382586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698793/real 1787698793] req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382599:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382607:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698793.382614:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698793.382616:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698793.382620:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382627:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698793.382632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382638:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698793.382649:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382654:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698793.382664:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698793.382670:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698818 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382678:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698793.382683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698793/real 1787698793] req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382692:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698793.382700:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698793.382706:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698793.382709:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698793.382712:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698793.382717:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698793.382723:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126382976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698793.382733:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698793.382742:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126383104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698793.382748:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698793.382752:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126383232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698793.382758:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698793.382763:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126383360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698793.382769:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698793.382773:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126383488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698818 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698793.382778:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698793.893509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698793.893509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698793.893509:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698794.021532:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698794.917511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698794.917511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698794.917511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698795.045580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698795.301517:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301368687, at 4301368704 00000100:00000040:3.0:1787698795.301517:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301368687, at 4301368704 00000100:00100000:1.0:1787698795.941482:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698795.941492:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698795.941503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698796.069646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698796.965562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698796.965567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698796.965684:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698797.093597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698797.989509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698797.989549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698797.989556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698798.117608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698798.501587:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698798.501597:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698798.501601:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698798.501610:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6699 00000100:00080000:3.0:1787698798.501622:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698798.501625:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698798.501631:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501634:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698798.501639:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698798.501641:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698798.501643:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501646:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698798.501648:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501663:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698798.501691:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698798.501702:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698798.501741:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698798.501744:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698798.501761:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698798.501766:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698798.501767:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6699 00000100:00080000:3.0:1787698798.501778:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698798.501782:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698798.501786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126383616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698798.501787:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501792:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698798.501795:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698798.501797:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698798.501799:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501801:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698798.501802:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698798.501811:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698798.501815:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698798.501819:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698798.501825:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698798.501834:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698798.501841:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698798.501848:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698798.501851:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698798.501852:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698798.501855:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6699 00000100:00080000:3.0:1787698798.501857:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698798.501859:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698798.501861:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501862:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698798.501864:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698798.501872:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698798.501878:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698798.501878:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501883:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698798.501887:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698798.501892:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698798.501897:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698798.501898:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698798.501912:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698798.501922:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698798.501928:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698798.501930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698798/real 1787698798] req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698798.501933:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698798.501937:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698798.501945:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6699 00000100:00080000:3.0:1787698798.501949:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698798.501951:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698798.501956:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698798.501956:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501960:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698798.501965:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698798.501971:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698798.501972:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698798.501980:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.501986:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698798.501987:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698798.501992:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:1.0:1787698798.501994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698798.502001:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698798.502003:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698798.502010:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698798.502014:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126383616:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698798.502021:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698798.502025:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126383616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698798.502056:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698798.502059:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698798.502066:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698798.502069:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698798.502070:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698798.502073:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6699 00000100:00080000:3.0:1787698798.502076:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698798.502078:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698798.502082:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698798.502084:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.502088:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698798.502092:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698798.502096:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698798.502100:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698798.502102:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698798.502103:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698798.502106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383744:192.168.201.136@tcp:38:kworker.0 00000008:00100000:3.0:1787698798.502107:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698798.502114:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698798.502122:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698798.502128:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698798.502135:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502141:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698798.502152:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698798.502154:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6709) 00000100:00000040:1.0:1787698798.502167:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502180:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698798.502188:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698798/real 1787698798] req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502200:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698798.502214:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698798.502216:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698798.502220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698798.502235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502240:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698798.502249:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502254:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698798.502262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698798.502268:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502277:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698798.502282:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698798/real 1787698798] req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698798.502317:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698798.502319:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698798.502322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502331:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126383872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698798.502335:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502340:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698798.502350:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502355:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698798.502363:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698798.502367:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502378:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698798.502383:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698798/real 1787698798] req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698798.502410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698798.502413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698798.502445:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698798.502462:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126383744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698798.502469:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698798.502474:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126383872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698798.502479:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698798.502482:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126384000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698798.502486:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698798.502491:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502496:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698798.502504:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502515:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698798.502524:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698798.502529:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698823 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502534:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698798.502538:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698798/real 1787698798] req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502545:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698798.502550:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698798.502556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698798.502558:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698798.502560:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698798.502564:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698798.502567:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126384128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698823 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698798.502570:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698799.013541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698799.013542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698799.013542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698799.141549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698800.037531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698800.037531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698800.037531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698800.165611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698800.421539:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301373807, at 4301373824 00000100:00000040:3.0:1787698800.421539:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301373807, at 4301373824 00000100:00100000:3.0:1787698801.061545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698801.061548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698801.061675:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698801.189596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698802.085527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698802.085527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698802.085527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698802.213617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698803.109533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698803.109533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698803.109533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698803.237554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698803.621539:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698803.621547:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698803.621551:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698803.621558:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6704 00000100:00080000:3.0:1787698803.621567:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698803.621569:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.621571:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621574:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698803.621578:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698803.621581:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.621583:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621584:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698803.621587:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621598:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698803.621619:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698803.621642:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698803.621662:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698803.621666:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698803.621668:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698803.621672:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6704 00000100:00080000:3.0:1787698803.621675:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698803.621692:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.621694:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621696:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698803.621698:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698803.621701:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.621703:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621705:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698803.621706:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698803.621713:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698803.621716:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698803.621725:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698803.621739:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698803.621744:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698803.621794:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698803.621799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126384256:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698803.621801:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698803.621810:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698803.621818:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6704 00000100:00080000:3.0:1787698803.621824:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698803.621828:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.621830:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621831:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698803.621834:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698803.621836:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698803.621840:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698803.621844:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621854:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787698803.621860:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698803.621871:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698803.621877:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698803.621881:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698803.621891:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698803.621899:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698803.621905:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787698803.621912:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698803.621914:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698803.621919:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698803.621923:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6704 00000100:00080000:3.0:1787698803.621926:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698803.621929:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.621930:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621934:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698803.621936:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698803.621938:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.621941:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698803.621943:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698803.621946:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698803.621951:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698803.621967:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698803.621971:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698803.621978:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698803.621986:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.621998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698803/real 1787698803] req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698803.622000:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698803.622016:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698803.622022:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698803.622028:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6704 00000100:00080000:3.0:1787698803.622033:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698803.622035:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698803.622041:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698803.622058:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698803.622064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698803.622065:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698803.622071:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698803.622081:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698803.622084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698803.622091:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698803.622091:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698803.622100:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698803.622100:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698803.622114:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698803.622119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126384256:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698803.622124:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698803.622129:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126384256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698803.622131:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787698803.622149:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698803.622150:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698803.622161:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698803.622172:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6714) 00000100:00100000:1.0:1787698803.622183:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698803.622197:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622202:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698803.622211:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698803.622216:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698803.622227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698803/real 1787698803] req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622233:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698803.622243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698803.622244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698803.622246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622250:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698803.622254:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622258:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698803.622265:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622269:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698803.622274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698803.622277:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622283:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698803.622286:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698803/real 1787698803] req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622293:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698803.622303:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698803.622304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698803.622306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622310:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698803.622313:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698803.622325:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622329:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698803.622333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698803.622336:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622341:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698803.622344:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698803/real 1787698803] req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698803.622363:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698803.622365:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698803.622367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698803.622374:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126384384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698803.622378:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698803.622382:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126384512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698803.622387:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698803.622390:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126384640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698803.622394:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698803.622398:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622403:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698803.622411:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622415:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698803.622459:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698803.622464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698828 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622470:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698803.622473:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698803/real 1787698803] req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698803.622491:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698803.622496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698803.622498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698803.622499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698803.622504:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126384768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698803.622507:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126384768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698828 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698803.622512:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698804.133550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698804.133555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698804.133574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698804.261600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698805.157511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698805.157511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698805.157511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698805.285585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698805.541502:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301378927, at 4301378944 00000100:00000040:3.0:1787698805.541502:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301378927, at 4301378944 00000100:00100000:0.0:1787698806.181503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698806.181503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698806.181505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698806.309621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698807.205509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698807.205550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698807.205620:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698807.333599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698808.229566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698808.229567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698808.229568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698808.357671:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698808.741568:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698808.741575:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698808.741578:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698808.741586:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6709 00000100:00080000:2.0:1787698808.741595:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698808.741597:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741600:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741602:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698808.741606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698808.741608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741612:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698808.741614:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741626:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698808.741651:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698808.741664:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698808.741687:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698808.741691:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698808.741692:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698808.741697:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6709 00000100:00080000:2.0:1787698808.741700:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698808.741703:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741706:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741710:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698808.741712:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698808.741715:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741718:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741720:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698808.741726:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698808.741727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698808.741735:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698808.741741:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698808.741744:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126384896:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787698808.741755:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698808.741766:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698808.741774:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698808.741776:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698808.741780:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698808.741785:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698808.741793:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698808.741794:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6709 00000100:00080000:2.0:1787698808.741801:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698808.741804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741809:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698808.741811:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698808.741814:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741817:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741819:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698808.741822:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698808.741826:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698808.741831:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698808.741836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698808.741842:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698808.741859:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698808.741859:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698808.741887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698808.741891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698808.741892:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698808.741896:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6709 00000100:00080000:2.0:1787698808.741901:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698808.741904:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741907:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741909:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:1.0:1787698808.741911:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698808/real 1787698808] req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698808.741912:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698808.741920:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.741926:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.741928:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698808.741930:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698808.741936:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698808.741941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698808.741941:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698808.741951:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698808.741957:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698808.741959:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698808.741968:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698808.741975:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.741976:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.741993:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126384896:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698808.741997:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698808.742001:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126384896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698808.742005:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698808.742010:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:1.0:1787698808.742012:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:2.0:1787698808.742015:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6709 00000100:00080000:2.0:1787698808.742021:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698808.742024:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698808.742028:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698808.742030:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.742034:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698808.742039:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:1.0:1787698808.742042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698808.742046:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698808.742050:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698808.742053:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698808.742056:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698808.742059:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698808.742069:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:2.0:1787698808.742069:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698808.742075:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:1.0:1787698808.742080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698808.742087:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698808.742088:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698808.742116:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742118:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698808.742129:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698808/real 1787698808] req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698808.742131:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6719) 00000100:00100000:1.0:1787698808.742149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742156:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698808.742160:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698808.742162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698808.742164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742173:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698808.742176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698808.742189:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742194:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698808.742201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698808.742205:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742211:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698808.742214:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698808/real 1787698808] req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742221:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698808.742231:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698808.742232:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698808.742234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742238:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698808.742241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742247:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698808.742257:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742264:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698808.742274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698808.742280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742289:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698808.742294:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698808/real 1787698808] req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742302:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698808.742311:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698808.742313:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698808.742315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742318:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698808.742322:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126385024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698808.742327:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698808.742331:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126385152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698808.742335:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698808.742338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126385280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698808.742342:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698808.742346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742351:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698808.742358:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742362:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698808.742369:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698808.742373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698833 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742378:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698808.742381:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698808/real 1787698808] req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742387:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698808.742392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698808.742399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698808.742401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698808.742403:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698808.742407:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698808.742409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126385408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698833 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698808.742413:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698809.253586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698809.253586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698809.253593:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698809.381636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698810.277512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698810.277513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698810.277535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698810.405669:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698810.661538:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301384047, at 4301384064 00000100:00000040:3.0:1787698810.661538:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301384047, at 4301384064 00000100:00100000:3.0:1787698811.301553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698811.301554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698811.301598:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698811.429604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698812.325556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698812.325560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698812.325575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698812.453614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698813.349550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698813.349557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698813.349558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698813.477623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698813.861558:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698813.861568:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698813.861571:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698813.861584:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6714 00000100:00080000:3.0:1787698813.861595:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698813.861598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698813.861604:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698813.861609:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698813.861611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698813.861615:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698813.861618:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698813.861632:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698813.861659:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698813.861671:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698813.861690:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698813.861692:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698813.861709:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698813.861712:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6714 00000100:00080000:3.0:1787698813.861715:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698813.861717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698813.861722:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698813.861724:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698813.861727:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861729:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698813.861730:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698813.861732:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698813.861740:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698813.861743:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698813.861754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698813.861756:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698813.861772:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698813.861819:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698813.861824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126385536:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787698813.861826:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698813.861831:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698813.861841:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6714 00000100:00080000:3.0:1787698813.861845:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698813.861848:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861850:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698813.861852:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698813.861854:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698813.861862:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861867:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698813.861869:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698813.861871:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698813.861878:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698813.861891:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698813.861891:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698813.861899:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698813.861908:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698813.861915:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698813.861922:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698813.861924:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698813.861925:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698813.861929:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6714 00000100:00080000:3.0:1787698813.861934:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787698813.861941:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861943:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698813.861953:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698813.861957:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698813.861960:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.861966:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698813.861968:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698813.861970:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698813.861985:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698813.862001:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698813.862008:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698813.862032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698813/real 1787698813] req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698813.862033:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698813.862061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698813.862065:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698813.862103:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698813.862108:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698813.862108:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698813.862113:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698813.862119:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6714 00000100:00080000:3.0:1787698813.862126:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698813.862132:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.862138:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698813.862140:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00080000:1.0:1787698813.862141:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:3.0:1787698813.862143:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:1.0:1787698813.862152:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698813.862153:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698813.862164:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698813.862165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698813.862174:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698813.862181:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698813.862188:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126385536:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698813.862197:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698813.862198:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126385536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698813.862206:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698813.862218:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787698813.862220:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698813.862239:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698813.862247:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6724) 00000100:00000040:1.0:1787698813.862254:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862267:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698813.862282:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862289:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698813.862302:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698813.862309:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862325:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698813.862331:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698813/real 1787698813] req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862342:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698813.862358:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698813.862361:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698813.862365:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698813.862378:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862384:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698813.862393:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862398:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698813.862405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698813.862410:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862416:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698813.862454:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698813/real 1787698813] req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862479:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698813.862496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698813.862499:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698813.862503:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862511:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698813.862515:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862522:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698813.862531:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862536:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698813.862544:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698813.862548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862554:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698813.862557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698813/real 1787698813] req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862573:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698813.862577:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698813.862579:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698813.862581:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862585:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126385920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698813.862590:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862598:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698813.862609:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862616:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698813.862626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698813.862632:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698838 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862639:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698813.862643:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698813/real 1787698813] req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698813.862655:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698813.862662:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698813.862665:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698813.862669:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698813.862676:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698813.862682:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126385664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698813.862690:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698813.862697:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126385792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698813.862703:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698813.862707:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126385920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698813.862711:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698813.862714:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126386048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698838 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698813.862719:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698814.373572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698814.373573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698814.373574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698814.501617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698815.397545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698815.397545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698815.397545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698815.525648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698815.781538:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301389167, at 4301389184 00000100:00000040:2.0:1787698815.781540:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301389167, at 4301389184 00000100:00100000:1.0:1787698816.421559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698816.421559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698816.421560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698816.549645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698817.445499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698817.445499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698817.445578:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698817.573587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698818.469525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698818.469525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698818.469727:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698818.597562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698818.981546:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698818.981557:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698818.981562:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698818.981574:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6719 00000100:00080000:2.0:1787698818.981585:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698818.981589:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698818.981593:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698818.981596:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698818.981602:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698818.981605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698818.981608:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698818.981611:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698818.981614:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698818.981630:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698818.981659:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698818.981676:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698818.981704:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698818.981716:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698818.981718:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698818.981723:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6719 00000100:00080000:2.0:1787698818.981728:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698818.981734:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698818.981737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698818.981741:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00000040:1.0:1787698818.981745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698818.981746:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698818.981755:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698818.981762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126386176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698818.981763:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698818.981769:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698818.981773:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698818.981784:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698818.981786:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698818.981792:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698818.981797:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698818.981810:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698818.981822:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787698818.981830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698818.981840:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698818.981844:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698818.981846:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698818.981849:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698818.981855:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6719 00000100:00080000:2.0:1787698818.981864:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698818.981869:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698818.981873:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698818.981876:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698818.981878:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698818.981882:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698818.981886:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698818.981888:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00100000:1.0:1787698818.981890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698818/real 1787698818] req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698818.981896:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698818.981906:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698818.981913:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698818.981914:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698818.981924:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698818.981929:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787698818.981931:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787698818.981933:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698818.981939:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698818.981959:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126386176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698818.981971:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126386176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.981971:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698818.981992:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:2.0:1787698818.981996:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698818.982001:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698818.982003:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698818.982007:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698818.982012:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6719 00000100:00100000:1.0:1787698818.982018:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386304:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787698818.982021:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698818.982028:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698818.982031:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.982036:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698818.982043:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698818.982044:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698818.982049:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:1.0:1787698818.982054:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698818.982056:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698818.982062:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.982064:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698818.982071:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:1.0:1787698818.982076:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:2.0:1787698818.982077:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698818.982083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698818/real 1787698818] req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698818.982088:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698818.982092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698818.982098:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698818.982099:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698818.982107:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698818.982109:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698818.982113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698818.982117:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698818.982136:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698818.982145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.982153:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698818.982154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698818.982167:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698818.982171:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698818.982174:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698818.982180:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787698818.982184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787698818.982185:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698818.982190:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698818.982193:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6719 00000100:00080000:2.0:1787698818.982204:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698818.982205:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698818.982213:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698818/real 1787698818] req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.982213:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698818.982223:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698818.982228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698818.982230:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698818.982236:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698818.982239:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.982245:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698818.982248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698818.982254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698818.982254:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698818.982260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698818.982262:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698818.982270:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698818.982271:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698818.982280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126386304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698818.982283:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698818.982284:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698818.982291:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126386432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698818.982292:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:1.0:1787698818.982299:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698818.982305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698818.982312:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698818.982314:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698818.982328:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.982334:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698818.982337:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698818.982346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698818.982355:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698818.982357:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6730) 00000100:00000040:1.0:1787698818.982366:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698818.982370:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698818/real 1787698818] req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698818.982390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698818.982396:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698818.982401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698818.982403:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698818.982405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698818.982409:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698818.982412:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126386560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698818.982416:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698818.982443:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698818.982448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698818.982456:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698818.982460:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698818.982467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698818.982471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698843 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698818.982476:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698818.982479:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698818/real 1787698818] req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698818.982486:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698818.982491:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698818.982499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698818.982501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698818.982504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698818.982507:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698818.982510:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126386688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698843 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698818.982514:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698819.493472:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698819.493499:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698819.493502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698819.621603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698820.517529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698820.517529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698820.517529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698820.645554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698820.901516:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301394287, at 4301394304 00000100:00000040:3.0:1787698820.901517:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301394287, at 4301394304 00000100:00100000:1.0:1787698821.541517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698821.541517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698821.541605:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698821.669600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698822.565521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698822.565521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698822.565521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698822.693585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698823.589553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698823.589553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698823.589554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698823.717603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698824.101511:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698824.101521:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698824.101525:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698824.101537:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6725 00000100:00080000:2.0:1787698824.101547:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698824.101551:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101555:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101559:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698824.101563:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698824.101566:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101569:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101572:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698824.101575:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101588:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698824.101610:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698824.101633:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698824.101656:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698824.101659:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698824.101662:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698824.101666:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6725 00000100:00080000:2.0:1787698824.101669:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698824.101671:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101674:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101677:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698824.101680:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698824.101683:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101686:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101689:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698824.101695:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698824.101704:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698824.101708:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698824.101716:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698824.101728:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698824.101729:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698824.101801:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698824.101805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126386816:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698824.101806:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698824.101811:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698824.101821:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6725 00000100:00080000:2.0:1787698824.101824:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698824.101826:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101828:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101830:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698824.101832:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698824.101835:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101838:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101839:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698824.101840:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698824.101843:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698824.101848:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698824.101851:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698824.101861:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698824.101870:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101884:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698824.101890:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698824.101892:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698824.101893:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698824.101896:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6725 00000100:00080000:2.0:1787698824.101898:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698824.101900:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101902:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101903:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698824.101905:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698824.101907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.101909:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101910:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698824.101912:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698824.101916:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1787698824.101921:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698824.101925:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698824.101935:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698824.101943:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698824.101944:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698824.101983:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698824.101990:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698824.101993:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698824.101996:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6725 00000100:00080000:2.0:1787698824.101999:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698824.102002:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.102004:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.102006:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698824.102008:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698824.102012:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698824.102016:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698824.102019:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698824.102026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698824.102028:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698824.102040:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698824.102053:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698824.102063:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698824.102066:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698824.102125:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698824.102136:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6735) 00000100:00080000:1.0:1787698824.102155:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698824.102160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698824.102165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102171:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126386816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698824.102176:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126386816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698824.102184:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698824.102200:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102206:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698824.102218:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102223:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698824.102232:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698824.102237:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102248:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698824.102252:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698824.102270:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698824.102271:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698824.102273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102279:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126386944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698824.102285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102306:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698824.102319:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102325:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698824.102333:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698824.102338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102344:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698824.102348:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102355:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698824.102365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698824.102366:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698824.102368:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102372:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698824.102376:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698824.102389:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102393:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698824.102398:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698824.102408:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102414:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698824.102417:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102464:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102469:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698824.102474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698824.102475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698824.102477:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698824.102486:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698824.102498:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102502:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698824.102508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698824.102512:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698849 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102518:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698824.102521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698824/real 1787698824] req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698824.102533:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698824.102537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698824.102539:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698824.102541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698824.102545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698824.102548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126386944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698824.102552:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698824.102555:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126387072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698824.102560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698824.102562:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126387200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698824.102566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698824.102569:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126387328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698849 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698824.102573:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698824.613532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698824.613552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698824.613552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698824.741587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698825.637537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698825.637557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698825.637595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698825.765607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698826.021529:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301399407, at 4301399424 00000100:00000040:3.0:1787698826.021529:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301399407, at 4301399424 00000100:00100000:0.0:1787698826.661532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698826.661532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698826.661532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698826.789617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698827.685505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698827.685505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698827.685505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698827.813596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698828.709522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698828.709522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698828.709522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698828.837665:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698829.221536:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698829.221547:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698829.221550:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698829.221558:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6730 00000100:00080000:3.0:1787698829.221567:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698829.221569:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221580:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698829.221583:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698829.221586:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221588:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221590:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698829.221591:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221603:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698829.221624:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698829.221638:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698829.221662:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698829.221680:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698829.221682:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698829.221687:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6730 00000100:00080000:3.0:1787698829.221691:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698829.221695:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221698:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221700:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698829.221703:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698829.221707:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221714:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:1.0:1787698829.221716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698829.221719:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698829.221733:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787698829.221738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126387456:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698829.221741:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698829.221763:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698829.221769:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.221787:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221791:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698829.221802:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698829.221806:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698829.221808:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698829.221812:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6730 00000800:00000100:1.0:1787698829.221817:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698829.221818:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698829.221824:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221828:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698829.221832:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698829.221841:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698829.221845:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698829.221850:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221854:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221856:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698829.221860:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698829.221863:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698829/real 1787698829] req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698829.221866:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698829.221873:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698829.221881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698829.221886:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698829.221898:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698829.221902:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698829.221913:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698829.221915:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698829.221918:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698829.221919:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698829.221922:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698829.221924:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698829.221928:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6730 00000100:00080000:3.0:1787698829.221934:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698829.221938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126387456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698829.221939:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221945:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221947:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698829.221949:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698829.221953:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126387456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698829.221953:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.221957:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221960:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698829.221963:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698829.221970:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698829.221976:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:3.0:1787698829.221978:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698829.221988:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698829.221994:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222000:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698829.222002:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698829.222008:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698829.222011:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698829.222016:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6730 00000100:00080000:3.0:1787698829.222019:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698829.222020:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.222022:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.222023:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698829.222024:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698829.222028:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698829.222030:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698829.222031:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698829.222032:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698829.222037:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387584:192.168.201.136@tcp:38:kworker.0 00000008:00100000:3.0:1787698829.222037:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698829.222042:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698829.222050:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698829.222056:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222060:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698829.222065:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6740) 00000100:00000040:1.0:1787698829.222072:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698829.222087:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698829.222095:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222105:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698829.222110:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698829/real 1787698829] req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698829.222137:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698829.222140:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698829.222144:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698829.222163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222172:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698829.222183:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222190:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698829.222213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698829.222220:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222230:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698829.222236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698829/real 1787698829] req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222247:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222255:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698829.222263:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698829.222266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698829.222269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698829.222283:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126387584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698829.222290:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698829.222296:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126387712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698829.222303:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698829.222310:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222318:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698829.222330:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222338:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698829.222349:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698829.222356:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222365:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698829.222371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698829/real 1787698829] req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698829.222397:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698829.222401:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698829.222404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222412:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698829.222417:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698829.222459:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222466:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698829.222476:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698829.222483:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698854 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222492:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698829.222497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698829/real 1787698829] req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222507:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698829.222515:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698829.222521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698829.222524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698829.222527:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698829.222534:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126387968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698829.222538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126387840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698829.222544:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698829.222550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126387968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698854 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698829.222556:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698829.733489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698829.733489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698829.733512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698829.861584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698830.757535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698830.757535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698830.757535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698830.885842:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698831.141535:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301404527, at 4301404544 00000100:00000040:3.0:1787698831.141535:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301404527, at 4301404544 00000100:00100000:0.0:1787698831.781539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698831.781539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698831.781539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698831.909623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698832.805487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698832.805492:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698832.805495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698832.933561:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698833.829485:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698833.829485:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698833.829496:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698833.957597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698834.341707:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698834.341717:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698834.341721:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698834.341731:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6735 00000100:00080000:1.0:1787698834.341742:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698834.341745:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.341748:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.341751:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698834.341756:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698834.341759:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.341760:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.341763:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698834.341766:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698834.341784:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698834.341814:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698834.341826:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698834.341846:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698834.341849:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698834.341850:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698834.341853:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6735 00000100:00080000:1.0:1787698834.341855:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698834.341857:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.341859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.341860:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698834.341862:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698834.341864:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.341866:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.341868:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698834.341869:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698834.341878:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698834.341882:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698834.341891:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698834.341896:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698834.341912:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698834.341921:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698834.341925:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787698834.341925:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126388096:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698834.341926:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698834.341933:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6735 00000100:00080000:1.0:1787698834.341938:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698834.341941:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.341943:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.341944:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698834.341946:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698834.341947:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.341948:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.341949:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698834.341951:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698834.341953:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698834.341956:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698834.341962:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698834.341968:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.341974:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698834.341975:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698834.341981:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698834.341984:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698834.341991:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6735 00000100:00080000:1.0:1787698834.341995:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698834.341999:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698834.342000:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.342004:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.342005:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698834.342007:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698834.342009:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.342011:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.342013:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698834.342014:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698834.342018:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698834.342021:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698834.342025:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698834.342030:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698834.342035:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698834.342038:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698834.342039:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698834.342041:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6735 00000100:00080000:1.0:1787698834.342043:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698834.342047:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698834.342048:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.342052:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.342053:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698834.342055:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698834.342057:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698834.342059:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698834.342061:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698834.342063:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698834.342065:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698834.342068:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698834.342075:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698834.342076:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698834.342094:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698834.342103:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6745) 00000100:00100000:2.0:1787698834.342156:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698834/real 1787698834] req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698834.342202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698834.342207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698834.342211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342219:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126388096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698834.342226:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126388096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698834.342237:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698834.342253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342263:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698834.342280:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342296:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698834.342313:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698834.342322:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342332:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698834.342338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698834/real 1787698834] req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698834.342366:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698834.342370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698834.342374:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342381:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698834.342387:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698834.342407:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342415:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698834.342458:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698834.342465:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342475:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698834.342481:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698834/real 1787698834] req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342492:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342510:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698834.342521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698834.342524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698834.342528:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342536:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698834.342542:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698834.342565:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342572:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698834.342583:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698834.342590:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342599:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698834.342605:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698834/real 1787698834] req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342623:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698834.342630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698834.342633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698834.342637:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342644:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698834.342649:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342656:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698834.342667:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342674:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698834.342683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698834.342690:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698859 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342706:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698834.342711:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698834/real 1787698834] req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342722:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698834.342731:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698834.342738:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698834.342741:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698834.342745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698834.342752:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698834.342759:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126388224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698834.342767:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698834.342773:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126388352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698834.342780:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698834.342785:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126388480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698834.342793:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698834.342799:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126388608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698859 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698834.342806:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698834.853532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698834.853532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698834.853532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698834.981533:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698835.877561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698835.877569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698835.877672:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698836.005557:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698836.261543:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301409647, at 4301409664 00000100:00000040:3.0:1787698836.261543:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301409647, at 4301409664 00000100:00100000:1.0:1787698836.901488:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698836.901492:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698836.901518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698837.029645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698837.925547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698837.925547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698837.925549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698838.053549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698838.949521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698838.949542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698838.949557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698839.077584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698839.461518:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698839.461528:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698839.461532:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698839.461541:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6740 00000100:00080000:1.0:1787698839.461552:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698839.461554:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461559:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698839.461562:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698839.461568:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698839.461571:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461574:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698839.461576:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698839.461580:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698839.461592:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698839.461617:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698839.461630:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698839.461653:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698839.461657:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698839.461659:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698839.461663:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6740 00000100:00080000:1.0:1787698839.461666:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698839.461668:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461669:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698839.461671:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698839.461674:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698839.461677:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461679:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698839.461680:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698839.461682:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698839.461697:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698839.461700:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698839.461709:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698839.461724:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.461729:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698839.461733:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698839.461748:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698839.461752:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698839.461758:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6740 00000100:00080000:1.0:1787698839.461765:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698839.461768:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461778:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698839.461778:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126388736:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787698839.461787:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698839.461792:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698839.461795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698839.461801:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698839.461804:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698839.461810:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698839.461816:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698839.461825:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698839.461828:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698839.461865:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461876:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698839.461888:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698839.461891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698839.461893:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698839.461898:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6740 00000100:00080000:1.0:1787698839.461903:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698839.461907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461910:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:2.0:1787698839.461913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698839.461915:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698839.461920:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698839.461925:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.461930:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698839.461934:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698839.461935:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698839.461949:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698839.461957:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698839.461964:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698839.461980:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698839.461982:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698839/real 1787698839] req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462006:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698839.462014:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462041:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698839.462044:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698839.462056:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698839.462065:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698839.462070:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698839.462075:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6740 00000100:00080000:2.0:1787698839.462079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:1.0:1787698839.462087:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698839.462088:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698839.462101:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698839.462102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126388736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698839.462117:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698839.462119:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126388736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698839.462131:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698839.462137:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:1.0:1787698839.462138:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698839.462161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698839.462162:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698839.462178:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698839.462180:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388864:192.168.201.136@tcp:38:kworker.0 00000020:00000040:1.0:1787698839.462190:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698839.462198:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698839.462203:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698839.462206:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698839.462212:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698839.462220:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:2.0:1787698839.462227:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698839.462235:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698839.462237:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698839.462252:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698839.462264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698839/real 1787698839] req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698839.462266:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462290:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462302:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698839.462306:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6750) 00000100:00080000:2.0:1787698839.462308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698839.462315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698839.462320:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462330:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698839.462336:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462342:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698839.462350:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462355:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698839.462361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698839.462365:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462371:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698839.462374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698839/real 1787698839] req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462381:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462386:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698839.462391:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698839.462392:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698839.462394:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462398:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126388992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698839.462402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462408:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698839.462415:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462448:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698839.462455:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698839.462459:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462465:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698839.462468:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698839/real 1787698839] req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462481:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698839.462489:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698839.462492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698839.462496:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462503:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698839.462509:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126388864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698839.462518:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698839.462524:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126388992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698839.462530:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698839.462534:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126389120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698839.462538:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698839.462543:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462549:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698839.462557:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462562:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698839.462570:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698839.462574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698864 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462580:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698839.462583:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698839/real 1787698839] req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462590:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698839.462595:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698839.462604:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698839.462606:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698839.462608:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698839.462612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698839.462615:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126389248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698864 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698839.462619:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698839.973572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698839.973580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698839.973850:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698840.101570:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698840.997541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698840.997549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698840.997552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698841.125609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698841.381546:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301414767, at 4301414784 00000100:00000040:2.0:1787698841.381550:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301414767, at 4301414784 00000100:00100000:0.0:1787698842.021560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698842.021560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698842.021565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698842.149614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698843.045536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698843.045550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698843.045552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698843.173589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698844.069518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698844.069518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698844.069581:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698844.197602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698844.581544:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698844.581555:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698844.581559:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698844.581570:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6745 00000100:00080000:3.0:1787698844.581583:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698844.581587:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581592:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581596:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698844.581602:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698844.581606:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581613:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698844.581617:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581635:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698844.581667:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698844.581694:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698844.581712:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698844.581716:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698844.581717:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698844.581722:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6745 00000100:00080000:3.0:1787698844.581726:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698844.581729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581732:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581734:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698844.581737:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698844.581741:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581743:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581745:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698844.581748:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698844.581759:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698844.581761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698844.581764:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698844.581774:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698844.581777:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126389376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698844.581783:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698844.581790:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698844.581793:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698844.581795:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698844.581803:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6745 00000100:00080000:3.0:1787698844.581806:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698844.581807:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698844.581809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581812:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581814:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:2.0:1787698844.581816:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581817:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698844.581819:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581822:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581823:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698844.581825:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698844.581830:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698844.581834:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698844.581842:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698844.581850:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698844.581855:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698844.581857:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698844.581860:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698844.581861:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698844.581864:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6745 00000100:00080000:3.0:1787698844.581867:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698844.581869:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581871:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698844.581873:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698844.581873:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698844.581875:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698844.581878:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581885:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581887:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698844.581889:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698844.581894:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698844.581897:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698844.581905:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698844/real 1787698844] req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698844.581905:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698844.581915:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698844.581922:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698844.581923:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698844.581925:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698844.581927:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698844.581930:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6745 00000100:00080000:3.0:1787698844.581933:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698844.581936:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581952:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581954:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698844.581956:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698844.581958:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698844.581961:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698844.581963:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787698844.581965:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698844.581965:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698844.581970:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698844.581974:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787698844.581975:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698844.581979:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:3.0:1787698844.581981:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698844.581983:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.581992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126389376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698844.581995:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582000:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126389376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698844.582005:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6755) 00000020:00000040:2.0:1787698844.582010:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698844.582026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698844.582048:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582053:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698844.582062:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698844.582066:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582073:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698844.582077:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698844/real 1787698844] req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582084:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698844.582094:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698844.582096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698844.582098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698844.582105:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698844.582117:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582121:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698844.582126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698844.582130:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582135:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698844.582138:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698844/real 1787698844] req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582149:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698844.582158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698844.582160:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698844.582162:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582165:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698844.582168:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698844.582180:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582184:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698844.582189:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698844.582192:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582198:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698844.582201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698844/real 1787698844] req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582211:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698844.582220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698844.582221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698844.582223:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582227:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698844.582230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582235:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698844.582241:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582245:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698844.582249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698844.582253:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698869 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582258:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698844.582268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698844/real 1787698844] req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582278:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698844.582283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698844.582287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698844.582289:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698844.582291:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698844.582295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126389888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698844.582298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126389504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698844.582303:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698844.582307:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126389632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698844.582311:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698844.582314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126389760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698844.582318:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698844.582320:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126389888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698869 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698844.582324:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698845.093532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698845.093532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698845.093532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698845.221603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698846.117536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698846.117536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698846.117536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698846.245591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698846.501494:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301419887, at 4301419904 00000100:00000040:3.0:1787698846.501494:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301419887, at 4301419904 00000100:00100000:0.0:1787698847.141552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698847.141552:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698847.141552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698847.269643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698848.165555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698848.165555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698848.165660:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698848.293577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698849.189506:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698849.189506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698849.189511:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698849.317627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698849.701562:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698849.701577:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698849.701581:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698849.701594:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6750 00000100:00080000:3.0:1787698849.701609:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698849.701613:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.701618:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.701623:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698849.701629:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698849.701633:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.701636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.701639:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698849.701643:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698849.701662:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698849.701694:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698849.701711:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698849.701737:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698849.701759:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698849.701762:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698849.701767:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6750 00000100:00080000:3.0:1787698849.701771:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698849.701774:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.701777:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.701779:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698849.701782:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698849.701784:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.701787:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.701790:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698849.701792:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698849.701803:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698849.701808:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698849.701819:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698849.701829:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698849.701840:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698849.701844:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698849.701847:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698849.701861:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698849.701871:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6750 00000100:00080000:3.0:1787698849.701879:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698849.701883:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.701888:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698849.701890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126390016:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787698849.701899:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698849.701903:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698849.701907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.701909:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.701911:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698849.701914:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698849.701920:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698849.701924:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698849.701933:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698849.701940:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698849.701961:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698849.701966:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698849.701978:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698849.701982:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698849.701984:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698849.701988:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6750 00000100:00080000:3.0:1787698849.702003:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698849.702006:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.702008:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.702009:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698849.702011:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698849.702014:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.702019:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:2.0:1787698849.702020:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698849.702021:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698849.702027:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698849.702034:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698849.702045:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698849.702051:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698849.702059:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698849.702072:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698849.702081:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698849.702084:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698849.702086:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698849.702090:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6750 00000100:00080000:3.0:1787698849.702095:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698849.702098:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.702101:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.702103:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698849.702106:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698849.702109:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698849.702112:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698849.702115:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787698849.702118:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698849/real 1787698849] req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698849.702121:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698849.702130:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698849.702137:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698849.702145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698849.702148:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698849.702176:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698849.702176:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698849.702206:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698849.702207:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6760) 00000100:00080000:2.0:1787698849.702213:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698849.702220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702231:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126390016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698849.702239:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126390016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698849.702257:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698849.702274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698849.702297:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702305:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698849.702316:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698849.702324:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702334:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698849.702339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698849/real 1787698849] req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702358:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698849.702365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698849.702369:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698849.702372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698849.702387:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702395:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698849.702408:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702415:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698849.702451:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698849.702458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702466:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698849.702472:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698849/real 1787698849] req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702482:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702490:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698849.702498:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698849.702501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698849.702505:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702513:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698849.702519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702528:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698849.702541:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702557:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698849.702571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698849.702578:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702588:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698849.702594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698849/real 1787698849] req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702605:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698849.702621:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698849.702623:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698849.702627:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698849.702640:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702649:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698849.702662:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702668:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698849.702679:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698849.702685:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698874 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702694:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698849.702699:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698849/real 1787698849] req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702711:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698849.702719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698849.702727:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698849.702785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698849.702790:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698849.702795:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698849.702799:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126390144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698849.702805:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698849.702809:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126390272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698849.702818:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698849.702821:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126390400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698849.702825:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698849.702827:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126390528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698874 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698849.702831:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698850.213547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698850.213547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698850.213547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698850.341586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698851.237529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698851.237529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698851.237537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698851.365613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698851.621537:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301425007, at 4301425024 00000100:00000040:3.0:1787698851.621537:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301425007, at 4301425024 00000100:00100000:0.0:1787698852.261537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698852.261546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698852.261562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698852.389538:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698853.285532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698853.285533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698853.285560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698853.413577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698854.309517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698854.309566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698854.309838:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698854.437602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698854.821521:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698854.821531:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698854.821536:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698854.821545:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6755 00000100:00080000:0.0:1787698854.821555:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698854.821557:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821560:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821562:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698854.821566:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698854.821569:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821570:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821572:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698854.821575:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821585:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698854.821606:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698854.821617:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698854.821638:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698854.821642:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698854.821644:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698854.821648:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6755 00000100:00080000:0.0:1787698854.821652:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698854.821655:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821659:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821661:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698854.821663:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698854.821666:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821668:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821669:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698854.821671:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698854.821679:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698854.821682:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698854.821687:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698854.821696:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698854.821708:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126390656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698854.821712:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698854.821724:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698854.821727:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698854.821729:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698854.821733:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6755 00000100:00080000:0.0:1787698854.821738:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698854.821742:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698854.821745:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821753:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821777:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:2.0:1787698854.821777:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821782:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698854.821788:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821791:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821795:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698854.821798:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698854.821803:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698854.821809:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698854.821819:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698854.821830:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698854.821838:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698854.821840:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698854.821842:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698854.821846:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6755 00000100:00080000:0.0:1787698854.821850:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698854.821853:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698854.821860:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698854.821861:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698854.821866:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698854.821874:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698854.821882:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390784:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787698854.821888:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698854.821893:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698854.821898:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698854.821902:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698854.821907:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698854.821910:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698854.821924:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698854.821926:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:2.0:1787698854.821935:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698854.821951:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698854.821957:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390912:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787698854.821965:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698854.821969:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698854.821971:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698854.821973:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698854.821976:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6755 00000100:00080000:0.0:1787698854.821983:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698854.821987:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821991:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.821994:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.821996:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698854.821997:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698854.821998:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:0.0:1787698854.822000:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698854.822002:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698854.822003:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698854.822006:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698854.822013:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698854.822016:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698854.822020:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698854.822029:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698854.822036:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787698854.822050:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698854.822057:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698854.822067:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698854.822069:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6765) 00000100:00000040:2.0:1787698854.822079:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:2.0:1787698854.822092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698854.822100:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698854.822113:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698854.822120:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698854.822132:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698855.333533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698855.333533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698855.333540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698855.461602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698856.357500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698856.357501:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698856.357542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698856.485620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698856.741580:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301430127, at 4301430144 00000100:00000040:2.0:1787698856.741588:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301430127, at 4301430144 00000100:00100000:0.0:1787698857.381558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698857.381558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698857.381558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698857.509615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698857.893782:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698857.893849:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698857.893869:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698857.893922:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698857.893942:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698857.893953:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698857.893964:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698857.893974:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698857.893984:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698857.893993:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698857.894004:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698857.894024:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698879 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698854/real 1787698857] req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894072:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698857.894081:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698857.894091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698857.894096:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698857.894099:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391040:192.168.201.136@tcp:8:kworker.0 00000100:00100000:2.0:1787698857.894114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698854/real 1787698857] req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894124:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698857.894132:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698857.894139:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698857.894142:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698857.894145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894156:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390784:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787698857.894163:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698854/real 1787698857] req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894173:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698857.894180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698857.894188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698857.894191:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698857.894194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894200:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126390912:192.168.201.136@tcp:38:kworker.0 00000100:00100000:2.0:1787698857.894208:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698854/real 1787698857] req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698857.894233:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698857.894241:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698857.894244:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698857.894248:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894255:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126390656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698857.894262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126391040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698857.894276:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698857.894288:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126390784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698857.894295:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698857.894302:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126390912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698857.894309:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698857.894314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126390656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698857.894320:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698857.894327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698854/real 1787698857] req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894338:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698857.894346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698857.894353:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698857.894356:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698857.894360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698857.894365:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698857.894369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126391168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698879 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698857.894374:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698858.405492:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698858.405522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698858.405525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698858.533659:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698859.429537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698859.429540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698859.429544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698859.557597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698859.941561:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698859.941570:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698859.941574:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698859.941592:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6760 00000100:00080000:3.0:1787698859.941622:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698859.941625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941630:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941635:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698859.941641:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698859.941645:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941647:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941651:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698859.941655:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941670:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698859.941729:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698859.941748:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698859.941777:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698859.941781:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698859.941783:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698859.941788:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6760 00000100:00080000:3.0:1787698859.941792:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698859.941794:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941797:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941799:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698859.941802:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698859.941805:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941810:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698859.941812:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698859.941831:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698859.941836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698859.941851:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698859.941866:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698859.941874:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698859.941877:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698859.941879:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698859.941882:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6760 00000100:00080000:3.0:1787698859.941886:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698859.941890:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941892:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941894:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698859.941896:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698859.941898:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941900:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941902:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698859.941905:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698859.941910:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698859.941914:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698859.941923:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698859.941932:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698859.941952:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698859.941955:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698859.941957:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698859.941961:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6760 00000100:00080000:3.0:1787698859.941965:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698859.941968:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941970:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941972:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698859.941975:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698859.941977:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.941980:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.941982:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698859.941985:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698859.941991:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698859.941995:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698859.942005:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698859.942015:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698859.942024:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698859.942026:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698859.942028:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698859.942032:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6760 00000100:00080000:3.0:1787698859.942036:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698859.942043:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.942046:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.942048:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698859.942050:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698859.942054:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698859.942056:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698859.942058:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698859.942060:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698859.942065:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698859.942070:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698859.942080:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698859.942088:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698859.942106:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6771) 00000100:00000040:2.0:1787698859.942192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.942208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126391296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698859.942246:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.942256:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698859.942304:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698859.942325:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.942363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698859/real 1787698859] req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.942382:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.942391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698859.942399:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698859.942404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698859.942409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.942418:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126391296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698859.943044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943054:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698859.943077:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943084:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698859.943110:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698859.943120:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943133:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698859.943139:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698859/real 1787698859] req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943151:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943159:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698859.943166:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698859.943170:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698859.943174:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943181:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698859.943186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943194:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698859.943206:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943213:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698859.943224:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698859.943230:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943237:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698859.943242:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698859/real 1787698859] req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698859.943267:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698859.943270:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698859.943273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698859.943287:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943296:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698859.943310:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943317:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698859.943329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698859.943335:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943344:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698859.943350:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698859/real 1787698859] req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943368:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698859.943375:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698859.943378:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698859.943382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943390:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698859.943396:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943406:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698859.943437:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943445:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698859.943457:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698859.943464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698884 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943473:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698859.943478:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698859/real 1787698859] req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943488:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698859.943495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698859.943501:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698859.943505:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698859.943508:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698859.943514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126391808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698859.943521:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126391296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698859.943530:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698859.943542:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126391424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698859.943549:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698859.943554:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126391552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698859.943560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698859.943564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126391680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698859.943569:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698859.943573:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126391808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698884 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698859.943579:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698860.453557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698860.453557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698860.453557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698860.581657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698861.477516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698861.477540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698861.477744:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698861.605611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698861.861507:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301435247, at 4301435264 00000100:00000040:3.0:1787698861.861508:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301435247, at 4301435264 00000100:00100000:3.0:1787698862.501477:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698862.501485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698862.501500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698862.629622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698863.525580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698863.525581:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698863.525581:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698863.653603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698864.549571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698864.549571:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698864.549571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698864.677635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698865.061522:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.061532:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698865.061537:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.061548:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6766 00000100:00080000:3.0:1787698865.061558:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698865.061561:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061564:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061589:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698865.061606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698865.061611:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061620:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698865.061624:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061638:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698865.061668:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698865.061680:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.061699:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.061701:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698865.061703:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.061706:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6766 00000100:00080000:3.0:1787698865.061709:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698865.061711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061713:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061715:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698865.061718:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698865.061720:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061722:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061724:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698865.061726:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698865.061736:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698865.061739:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698865.061751:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698865.061763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698865.061763:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.061790:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.061801:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698865.061803:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.061811:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6766 00000100:00080000:3.0:1787698865.061814:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698865.061820:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698865.061823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126391936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698865.061825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061830:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698865.061836:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698865.061839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061841:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061843:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698865.061844:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698865.061847:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698865.061851:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698865.061861:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698865.061867:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.061874:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698865.061876:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.061877:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698865.061883:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.061893:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6766 00000100:00080000:3.0:1787698865.061901:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698865.061904:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698865.061910:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061913:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061916:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698865.061918:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698865.061920:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061923:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698865.061926:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698865.061929:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698865.061932:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698865.061938:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698865.061945:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698865.061951:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698865.061953:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698865.061954:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698865.061964:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6766 00000800:00000100:2.0:1787698865.061965:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698865.061967:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698865.061970:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061972:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061974:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698865.061976:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698865.061982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698865.061984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698865.061986:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698865.061992:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698865.061994:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698865.061997:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698865.062005:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698865.062015:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698865.062023:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698865.062031:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6776) 00000100:00100000:2.0:1787698865.062051:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062081:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698865.062090:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698865.062095:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698865.062100:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062119:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126391936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698865.062127:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126391936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698865.062138:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698865.062157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062167:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698865.062183:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062191:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698865.062205:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698865.062212:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062228:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698865.062234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062245:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698865.062261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698865.062264:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698865.062269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062277:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698865.062283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062292:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698865.062304:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062311:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698865.062322:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698865.062328:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062338:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698865.062343:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062354:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698865.062370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698865.062373:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698865.062377:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698865.062391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698865.062412:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062455:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698865.062466:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698865.062473:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062482:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698865.062487:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062501:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698865.062515:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698865.062518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698865.062521:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698865.062533:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062540:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698865.062552:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062559:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698865.062568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698865.062574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698890 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062583:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698865.062588:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698865/real 1787698865] req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062615:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698865.062635:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698865.062642:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698865.062646:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698865.062649:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698865.062657:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698865.062662:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126392064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698865.062667:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698865.062671:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126392192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698865.062675:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698865.062678:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126392320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698865.062682:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698865.062685:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126392448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698890 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698865.062689:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698865.573489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698865.573504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698865.573532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698865.701605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698866.597460:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698866.597497:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698866.597506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698866.725607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698866.981529:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301440367, at 4301440384 00000100:00000040:2.0:1787698866.981532:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301440367, at 4301440384 00000100:00100000:1.0:1787698867.621538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698867.621538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698867.621576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698867.749588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698868.645487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698868.645505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698868.645553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698868.773663:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698869.669580:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698869.669588:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698869.669588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698869.797609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698870.181555:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698870.181563:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698870.181567:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698870.181575:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6771 00000100:00080000:1.0:1787698870.181584:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698870.181586:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181589:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181591:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698870.181595:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698870.181598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181600:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181603:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698870.181606:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181620:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698870.181645:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698870.181656:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698870.181672:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698870.181675:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698870.181676:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698870.181679:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6771 00000100:00080000:1.0:1787698870.181681:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698870.181683:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181685:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181687:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698870.181690:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698870.181692:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181694:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181695:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698870.181698:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698870.181705:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698870.181709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698870.181720:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698870.181721:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698870.181735:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698870.181745:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698870.181748:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787698870.181749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126392576:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698870.181750:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698870.181758:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6771 00000100:00080000:1.0:1787698870.181761:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698870.181764:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181768:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698870.181771:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698870.181774:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181777:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181779:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787698870.181783:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698870.181786:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698870.181795:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698870.181799:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181803:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698870.181815:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698870.181823:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698870.181829:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698870.181832:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698870.181833:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698870.181837:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6771 00000100:00080000:1.0:1787698870.181842:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698870.181842:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181847:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181849:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181851:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698870.181853:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698870.181855:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181857:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181858:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698870.181862:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698870.181863:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698870.181869:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698870.181875:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698870.181882:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698870.181888:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698870.181893:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698870.181896:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698870.181898:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698870.181901:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698870/real 1787698870] req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698870.181903:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6771 00000100:00080000:1.0:1787698870.181908:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698870.181913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181918:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181920:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698870.181922:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698870.181924:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698870.181925:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698870.181926:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698870.181928:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698870.181931:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698870.181934:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698870.181939:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698870.181940:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698870.181972:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698870.181974:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698870.181988:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6781) 00000100:00080000:2.0:1787698870.181991:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698870.181997:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698870.182002:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126392576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698870.182019:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126392576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698870.182029:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698870.182044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698870.182061:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182066:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698870.182075:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698870.182079:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182086:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698870.182090:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698870/real 1787698870] req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182102:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698870.182106:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698870.182108:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698870.182110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698870.182118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698870.182133:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182137:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698870.182142:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698870.182146:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182152:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698870.182154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698870/real 1787698870] req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698870.182169:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698870.182171:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698870.182173:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698870.182180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182184:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698870.182192:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182196:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698870.182201:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698870.182204:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182209:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698870.182212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698870/real 1787698870] req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182219:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182224:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698870.182227:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698870.182229:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698870.182231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126392960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698870.182237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182242:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698870.182248:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182256:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698870.182260:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698870.182264:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698895 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182268:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698870.182271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698870/real 1787698870] req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698870.182281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698870.182285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698870.182287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698870.182289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698870.182293:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698870.182296:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126392704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698870.182300:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698870.182303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126392832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698870.182306:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698870.182309:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126392960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698870.182313:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698870.182316:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126393088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698895 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698870.182320:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698870.693535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698870.693535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698870.693535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698870.821642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698871.717568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698871.717569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698871.717576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698871.845639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698872.101543:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301445487, at 4301445504 00000100:00000040:3.0:1787698872.101546:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301445487, at 4301445504 00000100:00100000:2.0:1787698872.741573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698872.741596:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698872.741597:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698872.869644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698873.765576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698873.765587:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698873.765588:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698873.893681:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698874.789592:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698874.789592:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698874.789594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698874.917608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698875.301537:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698875.301548:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698875.301552:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698875.301564:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6776 00000100:00080000:1.0:1787698875.301576:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698875.301578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301582:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301585:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698875.301590:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698875.301592:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301594:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301597:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698875.301599:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301613:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698875.301640:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698875.301653:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698875.301674:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698875.301678:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698875.301682:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698875.301686:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6776 00000100:00080000:1.0:1787698875.301688:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698875.301691:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301693:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301695:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698875.301697:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698875.301699:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301701:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301703:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698875.301705:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698875.301714:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698875.301717:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698875.301731:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698875.301731:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698875.301754:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698875.301762:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698875.301765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698875.301766:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698875.301772:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6776 00000100:00100000:2.0:1787698875.301773:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126393216:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698875.301774:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698875.301779:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301782:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301784:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698875.301786:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698875.301788:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301789:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301790:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698875.301792:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698875.301795:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698875.301799:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698875.301803:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698875.301813:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698875.301820:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698875.301820:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698875.301824:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698875.301827:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698875.301833:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6776 00000100:00080000:1.0:1787698875.301836:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698875.301839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301841:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301842:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698875.301843:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698875.301844:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301846:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301848:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787698875.301849:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698875.301852:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698875.301857:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698875.301859:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698875.301864:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698875.301870:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698875.301877:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698875.301880:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698875.301882:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698875.301886:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6776 00000100:00080000:1.0:1787698875.301890:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698875.301891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301898:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301902:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301905:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698875.301908:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698875.301912:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698875.301913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698875.301920:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698875.301922:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698875.301923:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698875.301926:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698875.301932:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698875.301940:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698875.301947:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698875.301955:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6786) 00000100:00100000:2.0:1787698875.301956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698875/real 1787698875] req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.301977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.301986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698875.301996:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698875.302000:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698875.302005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302012:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126393216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698875.302018:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126393216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698875.302028:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698875.302045:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302053:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698875.302067:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302074:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698875.302086:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698875.302093:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302102:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698875.302107:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698875/real 1787698875] req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302115:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302126:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698875.302134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698875.302136:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698875.302140:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302147:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698875.302152:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302161:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698875.302173:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302177:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698875.302185:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698875.302189:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302195:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698875.302198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698875/real 1787698875] req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302204:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698875.302215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698875.302217:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698875.302219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302223:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698875.302226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698875.302238:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302242:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698875.302247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698875.302250:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302255:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698875.302258:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698875/real 1787698875] req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302276:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698875.302283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698875.302286:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698875.302290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302297:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698875.302303:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302308:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698875.302317:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302322:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698875.302329:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698875.302333:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698900 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302342:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698875.302347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698875/real 1787698875] req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302358:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698875.302367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698875.302374:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698875.302377:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698875.302381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698875.302389:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698875.302403:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126393344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698875.302410:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698875.302414:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126393472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698875.302453:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698875.302458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126393600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698875.302462:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698875.302466:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126393728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698900 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698875.302470:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698875.813505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698875.813505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698875.813542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698875.941644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698876.837497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698876.837500:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698876.837589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698876.965575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698877.221561:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301450607, at 4301450624 00000100:00000040:2.0:1787698877.221670:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301450607, at 4301450624 00000100:00100000:0.0:1787698877.861544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698877.861544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698877.861545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698877.989620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698878.885544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698878.885550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698878.885557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698879.013631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698879.909525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698879.909525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698879.909525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698880.037630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698880.421588:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698880.421600:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698880.421605:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698880.421618:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6781 00000100:00080000:3.0:1787698880.421632:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698880.421636:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.421640:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.421644:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698880.421650:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698880.421661:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.421664:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.421667:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698880.421671:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698880.421689:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698880.421734:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698880.421758:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698880.421785:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698880.421789:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698880.421791:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698880.421796:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6781 00000100:00080000:3.0:1787698880.421799:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698880.421809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.421812:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698880.421835:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698880.421835:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698880.421839:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698880.421848:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.421855:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.421857:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698880.421862:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698880.421863:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126393856:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698880.421875:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698880.421882:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698880.421896:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698880.421903:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698880.421915:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.421921:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698880.421932:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698880.421936:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698880.421938:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698880.421944:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6781 00000100:00080000:3.0:1787698880.421957:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698880.421961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.421964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.421966:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000800:00000100:2.0:1787698880.421970:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698880.421971:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698880.421978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.421982:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.421984:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698880.421988:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698880.421993:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698880.421995:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698880.422008:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698880.422031:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698880.422044:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698880.422053:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698880.422056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698880/real 1787698880] req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698880.422058:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698880.422063:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698880.422072:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6781 00000100:00080000:3.0:1787698880.422078:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698880.422084:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.422087:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.422089:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:2.0:1787698880.422090:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698880.422092:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698880.422107:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698880.422111:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.422119:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.422122:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698880.422123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787698880.422127:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787698880.422131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:3.0:1787698880.422136:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698880.422138:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698880.422145:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698880.422154:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126393856:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698880.422161:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698880.422165:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126393856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698880.422191:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787698880.422199:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698880.422209:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698880.422210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698880.422224:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698880.422228:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787698880.422229:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393984:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698880.422237:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6781 00000100:00000040:2.0:1787698880.422248:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698880.422250:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698880.422258:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698880.422261:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.422266:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698880.422270:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698880.422276:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698880.422279:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698880.422282:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698880.422286:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698880.422289:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:2.0:1787698880.422295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:3.0:1787698880.422296:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698880.422303:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698880.422306:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698880.422317:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698880.422328:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787698880.422351:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698880/real 1787698880] req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698880.422383:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6791) 00000100:00100000:2.0:1787698880.422384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698880.422406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698880.422409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698880.422413:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126393984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698880.422456:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698880.422478:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422485:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698880.422495:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698880.422501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422510:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698880.422516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698880/real 1787698880] req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698880.422544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698880.422547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698880.422551:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422558:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698880.422564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126393984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698880.422572:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698880.422577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126394112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698880.422585:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698880.422592:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422601:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698880.422612:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422626:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698880.422635:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698880.422641:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422649:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698880.422655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698880/real 1787698880] req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422673:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698880.422681:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698880.422684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698880.422688:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698880.422701:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422709:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698880.422719:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422726:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698880.422735:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698880.422741:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698905 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422748:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698880.422754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698880/real 1787698880] req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422764:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698880.422772:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698880.422779:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698880.422782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698880.422786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698880.422794:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698880.422799:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126394240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698880.422806:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698880.422811:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126394368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698905 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698880.422818:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698880.933545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698880.933545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698880.933547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698881.061595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698881.957520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698881.957520:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698881.959838:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698882.085683:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698882.341541:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301455727, at 4301455744 00000100:00000040:3.0:1787698882.341541:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301455727, at 4301455744 00000100:00100000:0.0:1787698882.981568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698882.981568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698882.981575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698883.109711:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698884.005517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698884.005517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698884.005519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698884.133607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698885.029545:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698885.029545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698885.029545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698885.157605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698885.541541:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698885.541555:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698885.541559:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698885.541572:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6786 00000100:00080000:3.0:1787698885.541606:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698885.541611:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541615:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541619:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698885.541625:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698885.541628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541630:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541633:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698885.541636:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541649:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698885.541679:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698885.541704:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698885.541729:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698885.541733:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698885.541735:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698885.541740:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6786 00000100:00080000:3.0:1787698885.541743:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698885.541746:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541749:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541752:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698885.541755:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698885.541758:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541761:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541764:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698885.541767:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698885.541783:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698885.541786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698885.541789:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698885.541805:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698885.541809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126394496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698885.541825:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698885.541842:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698885.541845:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698885.541846:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698885.541851:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6786 00000100:00000040:2.0:1787698885.541852:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698885.541855:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698885.541859:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541863:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541867:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698885.541870:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698885.541874:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541877:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698885.541879:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541881:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698885.541884:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698885.541888:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698885.541891:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698885.541901:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698885.541910:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698885.541928:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698885.541936:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698885.541947:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:2.0:1787698885.541951:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698885.541951:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698885.541960:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6786 00000100:00080000:3.0:1787698885.541968:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698885.541971:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541976:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541978:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698885.541981:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698885.541984:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.541988:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.541989:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698885.541992:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698885.541996:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698885.541998:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698885/real 1787698885] req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698885.542001:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698885.542012:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698885.542025:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542026:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698885.542039:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698885.542044:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698885.542047:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698885.542050:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6786 00000100:00080000:3.0:1787698885.542054:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698885.542055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698885.542063:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.542068:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.542071:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698885.542073:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:2.0:1787698885.542075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698885.542076:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698885.542080:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698885.542083:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698885.542086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:3.0:1787698885.542087:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698885.542092:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698885.542096:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698885.542097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698885.542106:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698885.542112:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126394496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698885.542123:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698885.542135:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6796) 00000100:00000040:2.0:1787698885.542140:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126394496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698885.542154:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698885.542172:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542182:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698885.542198:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542206:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698885.542219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698885.542226:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542236:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698885.542256:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698885/real 1787698885] req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542268:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542276:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698885.542283:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698885.542292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698885.542295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698885.542307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698885.542334:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542341:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698885.542358:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698885.542366:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542376:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698885.542382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698885/real 1787698885] req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698885.542410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698885.542413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698885.542438:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542447:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698885.542453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698885.542476:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542483:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698885.542494:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698885.542501:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542510:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698885.542516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698885/real 1787698885] req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542527:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698885.542544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698885.542547:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698885.542551:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126394880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698885.542564:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542573:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698885.542585:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542598:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698885.542615:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698885.542622:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698910 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542631:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698885.542636:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698885/real 1787698885] req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542648:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698885.542657:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698885.542672:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698885.542676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698885.542680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698885.542688:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698885.542694:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126394624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698885.542702:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698885.542709:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126394752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698885.542716:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698885.542721:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126394880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698885.542729:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698885.542734:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126395008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698910 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698885.542740:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698886.053506:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698886.053508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698886.053549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698886.184034:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698887.077566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698887.077566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698887.077573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698887.205528:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698887.461503:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301460847, at 4301460864 00000100:00000040:2.0:1787698887.461527:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301460847, at 4301460864 00000100:00100000:3.0:1787698888.101494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698888.101521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698888.101577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698888.229744:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698889.125539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698889.125547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698889.125686:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698889.253572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698890.149568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698890.149570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698890.149706:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698890.277593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698890.661539:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698890.661549:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698890.661553:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698890.661563:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6791 00000100:00080000:1.0:1787698890.661579:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698890.661582:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661585:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661587:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698890.661591:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698890.661593:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661595:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661597:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698890.661599:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661612:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698890.661644:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698890.661658:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698890.661676:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698890.661679:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698890.661682:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698890.661685:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6791 00000100:00080000:1.0:1787698890.661688:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698890.661690:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661692:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661694:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698890.661696:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698890.661698:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661700:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661702:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698890.661704:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698890.661712:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698890.661715:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698890.661718:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698890.661730:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698890.661735:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126395136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698890.661743:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698890.661755:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698890.661757:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698890.661758:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698890.661760:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6791 00000100:00080000:1.0:1787698890.661762:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698890.661763:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661765:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661767:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698890.661768:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698890.661770:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661772:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661773:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698890.661776:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698890.661778:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698890.661783:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698890.661788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698890.661789:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698890.661798:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698890.661805:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698890.661810:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698890.661812:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698890.661813:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698890.661815:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6791 00000100:00080000:1.0:1787698890.661817:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698890.661819:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661821:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661822:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698890.661824:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698890.661825:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661827:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661828:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698890.661829:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698890.661832:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698890.661835:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000800:00000100:2.0:1787698890.661836:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698890.661842:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698890.661849:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698890.661854:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698890.661856:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698890.661864:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698890.661868:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698890.661874:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6791 00000100:00080000:1.0:1787698890.661876:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698890.661878:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661880:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661882:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698890.661883:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698890.661885:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698890.661887:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698890.661888:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698890.661890:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698890.661894:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698890.661897:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698890.661902:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698890/real 1787698890] req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698890.661904:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698890.661933:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698890.661940:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6801) 00000100:00100000:2.0:1787698890.661941:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.661952:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698890.661961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698890.661965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698890.661969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.661976:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126395136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698890.661982:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126395136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698890.661990:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698890.662051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662060:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698890.662084:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662089:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698890.662098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698890.662103:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662109:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698890.662113:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698890/real 1787698890] req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698890.662140:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698890.662143:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698890.662145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698890.662153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662171:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698890.662182:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662189:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698890.662200:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698890.662207:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662215:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698890.662219:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698890/real 1787698890] req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662225:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698890.662236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698890.662238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698890.662240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662244:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698890.662247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698890.662260:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662266:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698890.662284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698890.662291:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662300:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698890.662305:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698890/real 1787698890] req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662314:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662319:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698890.662324:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698890.662326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698890.662328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662332:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698890.662335:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662339:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698890.662347:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662351:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698890.662357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698890.662361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698915 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662366:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698890.662369:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698890/real 1787698890] req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662376:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698890.662381:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698890.662385:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698890.662386:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698890.662388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698890.662393:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698890.662396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126395264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698890.662400:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698890.662404:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126395392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698890.662409:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698890.662413:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126395520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698890.662445:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698890.662451:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126395648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698915 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698890.662458:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698891.173516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698891.173516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698891.173516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698891.301617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698892.197553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698892.197553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698892.197561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698892.325675:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698892.581545:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301465967, at 4301465984 00000100:00000040:2.0:1787698892.581558:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301465967, at 4301465984 00000100:00100000:1.0:1787698893.221530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698893.221532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698893.221532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698893.349623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698894.245542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698894.245542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698894.245543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698894.373572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698895.269490:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698895.269490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698895.269494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698895.399589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698895.781543:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698895.781557:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698895.781563:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698895.781577:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6796 00000100:00080000:1.0:1787698895.781591:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698895.781595:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781600:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781604:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698895.781609:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698895.781613:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781616:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781619:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698895.781621:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781636:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698895.781662:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698895.781673:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698895.781696:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698895.781699:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698895.781700:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698895.781703:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6796 00000100:00080000:1.0:1787698895.781706:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698895.781708:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781709:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781711:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698895.781713:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698895.781715:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781716:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781718:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698895.781719:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698895.781728:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698895.781736:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698895.781745:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698895.781746:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698895.781770:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698895.781780:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787698895.781780:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126395776:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698895.781793:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698895.781795:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698895.781800:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6796 00000100:00080000:1.0:1787698895.781805:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698895.781808:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781811:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781813:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:2.0:1787698895.781816:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698895.781818:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698895.781826:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698895.781831:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781836:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781840:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698895.781843:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698895.781848:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698895.781853:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698895.781865:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698895.781868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781882:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.781888:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698895.781899:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698895.781909:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698895.781912:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698895.781918:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6796 00000100:00080000:1.0:1787698895.781922:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698895.781933:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698895/real 1787698895] req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698895.781937:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.781946:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781950:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698895.781952:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:2.0:1787698895.781953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698895.781961:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698895.781967:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698895.781968:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.781974:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698895.781979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698895.781980:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787698895.781985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698895.781989:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698895.781990:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698895.781999:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698895.782002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126395776:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698895.782011:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698895.782017:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126395776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698895.782030:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698895.782033:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787698895.782044:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698895.782051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698895.782055:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698895.782059:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698895.782064:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6796 00000100:00100000:2.0:1787698895.782066:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395904:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787698895.782079:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698895.782084:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.782086:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.782089:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698895.782091:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698895.782093:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698895.782095:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698895.782097:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787698895.782101:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698895.782102:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698895.782110:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787698895.782113:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698895.782117:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:2.0:1787698895.782125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698895.782129:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698895.782134:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698895.782141:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698895.782160:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6806) 00000100:00000040:2.0:1787698895.782167:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698895.782177:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698895/real 1787698895] req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782187:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698895.782198:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698895.782199:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698895.782202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126395904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698895.782210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782215:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698895.782223:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782228:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698895.782235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698895.782239:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782245:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698895.782248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698895/real 1787698895] req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782255:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698895.782264:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698895.782266:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698895.782268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698895.782274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782280:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698895.782287:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782292:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698895.782298:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698895.782304:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782312:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698895.782317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698895/real 1787698895] req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782343:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698895.782348:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698895.782350:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698895.782353:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782357:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698895.782361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126395904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698895.782365:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698895.782369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126396032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698895.782374:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698895.782377:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126396160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698895.782381:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698895.782385:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782391:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698895.782404:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782411:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698895.782449:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698895.782455:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698920 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782461:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698895.782465:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698895/real 1787698895] req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782472:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698895.782477:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698895.782482:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698895.782483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698895.782485:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698895.782489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698895.782493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126396288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698920 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698895.782498:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698896.293537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698896.293537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698896.293543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698896.421569:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698897.317561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698897.317565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698897.317600:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698897.445626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698897.701546:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301471087, at 4301471104 00000100:00000040:3.0:1787698897.701546:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301471087, at 4301471104 00000100:00100000:0.0:1787698898.341594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698898.341595:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698898.342110:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698898.469562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698899.365540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698899.365546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698899.365546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698899.493595:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698900.389565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698900.389581:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698900.389711:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698900.517597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698900.901567:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698900.901575:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698900.901578:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698900.901586:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6801 00000100:00080000:2.0:1787698900.901595:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698900.901597:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901599:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901602:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698900.901606:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698900.901608:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901609:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901611:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698900.901613:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901626:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698900.901650:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698900.901662:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698900.901683:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698900.901685:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698900.901687:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698900.901690:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6801 00000100:00080000:2.0:1787698900.901692:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698900.901695:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901697:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901698:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698900.901700:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698900.901703:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901705:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901707:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698900.901709:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698900.901716:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698900.901720:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698900.901721:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698900.901731:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698900.901742:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.901746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126396416:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787698900.901751:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698900.901754:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698900.901761:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698900.901766:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6801 00000100:00080000:2.0:1787698900.901769:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698900.901771:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901772:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901774:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698900.901775:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698900.901779:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698900.901780:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901786:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901790:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698900.901794:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698900.901799:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698900.901801:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901804:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698900.901812:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698900.901819:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698900.901824:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698900.901826:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698900.901827:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698900.901830:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6801 00000100:00080000:2.0:1787698900.901832:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698900.901833:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901835:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901836:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698900.901838:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:1.0:1787698900.901843:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901843:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901848:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901849:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698900.901851:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698900.901854:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698900.901857:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698900.901862:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698900.901866:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698900.901883:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698900.901888:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698900.901890:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698900.901891:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698900.901893:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6801 00000100:00080000:2.0:1787698900.901895:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698900.901896:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901898:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901899:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698900.901900:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698900.901901:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698900.901903:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698900.901904:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698900.901906:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698900.901910:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698900.901913:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698900.901914:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698900/real 1787698900] req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698900.901921:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698900.901930:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.901932:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698900.901946:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6812) 00000100:00000040:1.0:1787698900.901950:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698900.901956:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698900.901959:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698900.901963:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.901968:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126396416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698900.901973:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126396416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698900.901979:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698900.902004:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902010:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698900.902021:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902026:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698900.902034:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698900.902039:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902047:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698900.902053:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698900/real 1787698900] req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902070:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902078:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698900.902086:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698900.902089:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698900.902092:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902097:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698900.902100:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902105:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698900.902115:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902119:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698900.902126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698900.902131:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902137:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698900.902140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698900/real 1787698900] req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902155:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698900.902159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698900.902161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698900.902163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902167:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698900.902169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698900.902181:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902185:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698900.902191:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698900.902194:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902199:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698900.902202:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698900/real 1787698900] req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902212:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698900.902221:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698900.902223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698900.902225:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902228:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698900.902232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698900.902243:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902247:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698900.902252:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698900.902255:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698925 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902261:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698900.902263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698900/real 1787698900] req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902269:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698900.902275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698900.902279:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698900.902280:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698900.902282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698900.902286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126396928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698900.902289:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126396544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698900.902293:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698900.902296:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126396672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698900.902300:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698900.902303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126396800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698900.902307:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698900.902309:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126396928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698925 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698900.902313:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698901.413533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698901.413533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698901.413636:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698901.541606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698902.437532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698902.437532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698902.437532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698902.565579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698902.821485:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301476207, at 4301476224 00000100:00000040:3.0:1787698902.821490:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301476207, at 4301476224 00000100:00100000:0.0:1787698903.461512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698903.461512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698903.461512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698903.589668:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698904.485524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698904.485524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698904.485553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698904.613563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698905.509494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698905.509494:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698905.509499:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698905.637621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698906.021543:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698906.021555:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698906.021559:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698906.021571:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6807 00000100:00080000:0.0:1787698906.021583:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698906.021587:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021592:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021596:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698906.021601:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698906.021605:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021608:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021611:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698906.021614:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021641:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698906.021667:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698906.021680:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.021703:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698906.021707:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698906.021709:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698906.021714:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6807 00000100:00080000:0.0:1787698906.021719:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698906.021722:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021729:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698906.021731:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698906.021735:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021737:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021739:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698906.021744:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698906.021746:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698906.021752:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698906.021758:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698906.021766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126397056:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698906.021768:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698906.021784:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.021793:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698906.021796:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698906.021798:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698906.021801:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6807 00000100:00080000:0.0:1787698906.021805:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698906.021807:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698906.021810:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021815:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021820:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698906.021823:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698906.021826:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698906.021835:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021838:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021842:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698906.021846:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698906.021854:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698906.021859:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698906.021866:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698906.021871:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698906.021884:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698906.021885:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.021916:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698906.021924:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698906.021926:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698906.021930:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6807 00000100:00080000:0.0:1787698906.021933:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698906.021937:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021940:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021942:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698906.021945:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698906.021949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698906.021950:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.021957:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.021960:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698906.021963:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698906.021967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698906.021970:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698906.021975:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698906.021977:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698906.021985:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698906.021990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787698906.021992:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698906.021994:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022022:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126397056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698906.022023:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022035:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126397056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698906.022052:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698906.022060:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698906.022063:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:1.0:1787698906.022068:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:0.0:1787698906.022069:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6807 00000100:00080000:0.0:1787698906.022075:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698906.022078:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698906.022081:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.022083:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:1.0:1787698906.022085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698906.022087:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698906.022094:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698906.022099:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698906.022101:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698906.022104:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698906.022108:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698906.022113:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698906.022113:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698906.022121:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698906.022122:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698906.022134:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698906.022139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698906.022146:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698906.022153:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022156:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698906.022164:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698906.022172:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6817) 00000100:00100000:1.0:1787698906.022174:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698906.022188:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698906.022192:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698906.022196:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022203:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698906.022211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022219:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698906.022231:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022236:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698906.022244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698906.022250:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022260:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698906.022265:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022276:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698906.022292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698906.022294:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698906.022297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022301:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698906.022304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022310:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698906.022320:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022327:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698906.022337:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698906.022344:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022353:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698906.022358:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022369:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698906.022380:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698906.022381:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698906.022383:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022387:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698906.022391:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126397184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698906.022396:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698906.022400:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126397312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698906.022404:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698906.022407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126397440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698906.022411:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698906.022416:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022440:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698906.022449:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022453:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698906.022461:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698906.022471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698931 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022477:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698906.022480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698906/real 1787698906] req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022487:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698906.022492:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698906.022496:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698906.022498:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698906.022500:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698906.022505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698906.022508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126397568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698931 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698906.022511:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698906.533522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698906.533522:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698906.533522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698906.661643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698907.557515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698907.557515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698907.557612:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698907.685614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698907.941539:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301481327, at 4301481344 00000100:00000040:3.0:1787698907.941539:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301481327, at 4301481344 00000100:00100000:1.0:1787698908.581526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698908.581526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698908.581535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698908.709545:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698909.605524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698909.605524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698909.605524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698909.733593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698910.629480:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698910.629480:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698910.629485:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698910.757611:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698911.141538:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698911.141548:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698911.141553:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698911.141563:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6812 00000100:00080000:0.0:1787698911.141574:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698911.141579:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141583:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141586:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698911.141591:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698911.141594:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141596:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141600:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698911.141603:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141618:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698911.141643:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698911.141657:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.141681:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698911.141685:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698911.141687:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698911.141690:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6812 00000100:00080000:0.0:1787698911.141694:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698911.141697:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141700:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141702:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698911.141705:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698911.141708:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141711:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141713:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698911.141716:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698911.141725:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787698911.141730:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698911.141731:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698911.141745:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698911.141749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126397696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698911.141765:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.141774:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698911.141778:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698911.141780:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698911.141785:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.141787:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6812 00000100:00080000:0.0:1787698911.141793:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698911.141796:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141799:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141802:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698911.141805:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698911.141826:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141837:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141841:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141843:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698911.141846:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698911.141852:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698911.141857:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:1.0:1787698911.141864:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787698911.141868:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698911.141880:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.141885:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.141894:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698911.141901:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698911.141903:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698911.141907:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6812 00000100:00080000:0.0:1787698911.141911:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698911.141914:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.141918:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141920:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698911.141923:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698911.141928:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698911.141929:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698911.141932:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.141937:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698911.141943:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698911.141950:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698911.141957:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698911.141967:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698911.141970:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698911.141993:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.141996:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698911.142014:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698911.142017:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698911.142018:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698911.142022:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698911.142026:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698911.142030:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6812 00000100:00080000:0.0:1787698911.142035:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698911.142039:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.142042:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698911.142044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698911.142045:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698911.142049:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698911.142056:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698911.142062:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698911.142066:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:1.0:1787698911.142067:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126397696:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698911.142071:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698911.142078:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698911.142081:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126397696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787698911.142084:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698911.142095:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:1.0:1787698911.142097:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698911.142109:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.142119:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698911.142121:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6822) 00000100:00100000:1.0:1787698911.142132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698911.142147:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.142155:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698911.142167:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698911.142175:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.142185:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698911.142191:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.142203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.142212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698911.142220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698911.142223:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698911.142227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.142235:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698911.142240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.142248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698911.143458:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143467:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698911.143488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698911.143497:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143507:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698911.143512:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143529:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143537:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698911.143544:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698911.143546:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698911.143550:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126397952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698911.143562:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143569:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698911.143582:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143588:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698911.143598:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698911.143604:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143612:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698911.143617:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143640:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698911.143646:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698911.143649:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698911.143653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143659:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698911.143665:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126397824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698911.143672:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698911.143678:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126397952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698911.143685:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698911.143690:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126398080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698911.143695:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698911.143705:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698911.143724:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143731:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698911.143740:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698911.143747:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698936 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143754:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698911.143759:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698911/real 1787698911] req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698911.143777:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698911.143784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698911.143787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698911.143791:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698911.143798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698911.143803:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126398208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698936 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698911.143824:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698911.653502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698911.653537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698911.653561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698911.781591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698912.677512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698912.677512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698912.677545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698912.805697:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698913.061531:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301486447, at 4301486464 00000100:00000040:3.0:1787698913.061534:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301486447, at 4301486464 00000100:00100000:0.0:1787698913.701569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698913.701569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698913.701574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698913.829622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698914.725503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698914.725554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698914.725568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698914.853575:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698915.749575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698915.749575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698915.749582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698915.878493:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698916.262816:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698916.262824:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698916.262827:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698916.262836:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6817 00000100:00080000:2.0:1787698916.262845:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698916.262847:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.262850:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.262853:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698916.262857:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698916.262860:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.262862:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.262865:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698916.262868:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698916.262878:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698916.262896:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698916.262908:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698916.262925:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698916.262929:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698916.262930:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698916.262932:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6817 00000100:00080000:2.0:1787698916.262935:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698916.262936:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.262938:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.262939:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698916.262940:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698916.262942:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.262944:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.262945:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698916.262946:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698916.262953:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698916.262956:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698916.263002:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698916.263013:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698916.263026:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126398336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698916.263036:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698916.263044:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698916.263046:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698916.263048:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698916.263050:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6817 00000100:00080000:2.0:1787698916.263053:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698916.263056:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.263058:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.263061:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698916.263063:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698916.263066:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.263068:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.263071:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698916.263081:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698916.263088:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787698916.263089:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698916.263096:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698916.263110:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787698916.263112:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698916.263123:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698916.263135:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698916.263138:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698916.263141:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698916.263146:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6817 00000100:00080000:2.0:1787698916.263150:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698916.263153:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.263155:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.263157:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698916.263159:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000800:00000100:1.0:1787698916.263165:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698916.263171:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.263176:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.263178:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698916.263180:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698916.263185:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698916.263190:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698916.263199:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698916.263208:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698916.263215:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698916.263218:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698916.263220:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698916.263223:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6817 00000100:00080000:2.0:1787698916.263226:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698916.263228:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.263231:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.263233:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698916.263235:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698916.263237:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698916.263240:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698916.263242:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698916.263244:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698916.263249:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698916.263253:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698916.263261:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698916.263268:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263276:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698916.263276:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6827) 00000100:00100000:1.0:1787698916.263314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698916/real 1787698916] req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698916.263340:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698916.263342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698916.263345:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126398336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698916.263355:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126398336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698916.263362:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698916.263375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698916.263409:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263442:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698916.263467:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698916.263472:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263479:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698916.263483:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698916/real 1787698916] req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263490:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698916.263499:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698916.263501:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698916.263504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698916.263512:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698916.263533:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263540:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698916.263550:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698916.263556:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263565:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698916.263570:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698916/real 1787698916] req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263579:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698916.263588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698916.263590:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698916.263592:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263601:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698916.263604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698916.263617:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263621:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698916.263628:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698916.263632:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263637:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698916.263640:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698916/real 1787698916] req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263647:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263652:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698916.263655:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698916.263657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698916.263660:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263664:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698916.263668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263672:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698916.263679:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263682:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698916.263687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698916.263691:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698941 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263695:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698916.263698:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698916/real 1787698916] req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263704:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698916.263709:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698916.263712:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698916.263714:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698916.263716:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698916.263719:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126398848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698916.263738:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126398464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698916.263747:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698916.263751:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126398592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698916.263755:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698916.263758:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126398720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698916.263761:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698916.263764:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126398848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698941 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698916.263767:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698916.773529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698916.773590:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698916.773619:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698916.901639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698917.797541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698917.797542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698917.797549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698917.925635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698918.181567:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301491568, at 4301491584 00000100:00000040:2.0:1787698918.181568:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301491568, at 4301491584 00000100:00100000:1.0:1787698918.821563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698918.821564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698918.821575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698918.949651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698919.845506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698919.845507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698919.845528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698919.973672:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698920.869569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698920.869570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698920.869710:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698920.999234:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698921.381570:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698921.381592:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698921.381598:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698921.381613:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6822 00000100:00080000:2.0:1787698921.381628:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698921.381632:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.381667:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381672:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698921.381678:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698921.381681:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.381684:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381687:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698921.381691:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381709:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698921.381737:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698921.381750:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698921.381773:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698921.381776:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698921.381777:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698921.381780:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6822 00000100:00080000:2.0:1787698921.381783:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698921.381785:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.381788:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381789:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698921.381791:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698921.381793:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.381794:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381796:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698921.381797:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698921.381806:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698921.381810:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698921.381816:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698921.381825:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698921.381833:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698921.381837:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698921.381838:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698921.381842:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6822 00000100:00080000:2.0:1787698921.381846:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698921.381857:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698921.381858:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698921.381865:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381877:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698921.381884:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698921.381899:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.381902:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381903:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698921.381907:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698921.381910:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126398976:192.168.201.136@tcp:250:kworker.0 00000008:00100000:2.0:1787698921.381914:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698921.381922:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698921.381935:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698921.381948:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698921.381959:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698921.381963:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698921.381965:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698921.381969:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698921.381978:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6822 00000100:00080000:2.0:1787698921.381983:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698921.381985:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.381987:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.381988:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698921.381991:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698921.381996:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698921.382005:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698921.382006:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.382015:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698921.382019:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698921.382024:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698921.382027:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698921.382034:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698921.382040:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698921.382049:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698921.382052:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698921.382054:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698921.382057:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6822 00000100:00080000:2.0:1787698921.382060:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698921.382061:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.382063:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.382064:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698921.382065:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698921.382067:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698921.382069:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698921.382079:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698921.382081:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698921.382083:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698921.382089:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698921.382109:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698921.382122:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698921.382127:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698921.382163:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6832) 00000100:00100000:1.0:1787698921.382170:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698921.382195:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698921.382204:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698921.382221:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:1.0:1787698921.382238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698921.382248:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698921.382260:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698921.382267:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698921.382281:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787698921.382288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698921.382298:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698921.382312:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698921.382320:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698921.382331:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:1.0:1787698921.382338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698921.382348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698921.382362:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698921.382369:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698921.382380:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787698921.893589:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698921.893589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698921.893590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698922.021670:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698922.917563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698922.917563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698922.917564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698923.045684:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698923.301537:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301496687, at 4301496704 00000100:00000040:2.0:1787698923.301542:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301496687, at 4301496704 00000100:00100000:0.0:1787698923.941578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698923.941582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698923.941715:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698924.069616:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1787698924.453740:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:0.0:1787698924.453782:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787698924.453798:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698924.453838:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698924.453855:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698924.453865:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698924.453875:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698924.453882:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698924.453891:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698924.453899:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787698924.453908:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:0.0:1787698924.453915:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698946 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.453988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698921/real 1787698924] req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454013:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698924.454518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698924.454528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698924.454532:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698924.454536:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399232:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698924.454558:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698921/real 1787698924] req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454574:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698924.454597:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698924.454605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698924.454609:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698924.454616:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454623:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399360:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698924.454631:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698921/real 1787698924] req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454642:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698924.454661:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698924.454673:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698924.454676:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698924.454679:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454690:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399488:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698924.454695:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698921/real 1787698924] req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454709:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698924.454717:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698924.454723:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698924.454727:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698924.454730:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.454738:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126398976:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698924.454744:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698921/real 1787698924] req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.455033:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698924.455044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698924.455051:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698924.455055:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698924.455059:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698924.455065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698924.455072:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126399232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698924.455086:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698924.455095:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126399360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698924.455101:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698924.455108:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126399488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698924.455114:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698924.455118:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126398976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698924.455124:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698924.455129:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126399104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698946 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698924.455135:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698924.965513:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698924.965513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698924.965513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698925.093629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698925.989533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698925.989533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698925.989701:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698926.117604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698926.501596:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698926.501609:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698926.501614:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698926.501628:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6827 00000100:00080000:0.0:1787698926.501642:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698926.501647:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.501652:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.501656:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698926.501663:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698926.501666:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.501668:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.501673:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698926.501677:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698926.501691:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698926.502397:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698926.502445:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698926.502477:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698926.502481:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698926.502483:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698926.502488:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6827 00000100:00080000:0.0:1787698926.502493:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698926.502496:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502500:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502502:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698926.502507:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698926.502510:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502513:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502515:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698926.502518:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698926.502530:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698926.502537:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698926.502561:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698926.502572:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698926.502588:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698926.502591:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698926.502594:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698926.502598:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6827 00000100:00080000:0.0:1787698926.502601:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698926.502605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502607:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502610:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698926.502612:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698926.502615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502617:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502620:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698926.502622:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698926.502627:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698926.502632:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698926.502642:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698926.502652:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698926.502661:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698926.502665:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698926.502667:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698926.502671:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6827 00000100:00080000:0.0:1787698926.502674:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698926.502678:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502680:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502683:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698926.502685:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698926.502687:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502690:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502692:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698926.502695:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698926.502700:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698926.502704:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698926.502713:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698926.502722:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698926.502731:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698926.502734:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698926.502736:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698926.502740:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6827 00000100:00080000:0.0:1787698926.502743:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698926.502746:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502748:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502750:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698926.502752:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698926.502755:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698926.502758:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698926.502760:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698926.502763:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698926.502767:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698926.502771:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698926.502779:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698926.502790:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698926.502801:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6837) 00000100:00000040:1.0:1787698926.503217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503234:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126399616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698926.503285:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503295:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698926.503347:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698926.503369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698926/real 1787698926] req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503462:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503473:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698926.503483:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698926.503487:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698926.503491:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503501:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126399616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698926.503509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503519:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698926.503533:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503540:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698926.503564:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698926.503572:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503584:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698926.503591:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698926/real 1787698926] req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503602:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503611:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698926.503619:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698926.503622:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698926.503626:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503633:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698926.503638:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503647:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698926.503661:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503669:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698926.503680:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698926.503687:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503695:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698926.503700:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698926/real 1787698926] req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698926.503736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698926.503739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698926.503744:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126399872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698926.503757:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503766:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698926.503778:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503785:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698926.503797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698926.503804:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503819:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698926.503824:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698926/real 1787698926] req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503842:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503851:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698926.503859:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698926.503862:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698926.503866:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503875:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698926.503882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503890:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698926.503901:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503908:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698926.503918:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698926.503925:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698951 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503935:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698926.503940:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698926/real 1787698926] req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503952:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698926.503960:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698926.503967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698926.503971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698926.503975:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698926.503983:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698926.503990:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126399616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698926.504002:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698926.504014:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126399744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698926.504020:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698926.504025:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126399872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698926.504033:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698926.504038:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126400000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698926.504045:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698926.504051:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126400128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698951 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698926.504058:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698927.013489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698927.013511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698927.013558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698927.141679:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698928.037514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698928.037514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698928.037601:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698928.165622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698928.421569:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301501808, at 4301501824 00000100:00000040:3.0:1787698928.421581:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301501808, at 4301501824 00000100:00100000:1.0:1787698929.061580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698929.061587:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698929.061590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698929.189663:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698930.085509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698930.085541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698930.085542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698930.213636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698931.109569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698931.109569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698931.109593:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698931.237615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698931.621542:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698931.621556:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698931.621561:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698931.621574:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6832 00000100:00080000:0.0:1787698931.621588:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698931.621591:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621596:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621600:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698931.621606:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698931.621609:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621612:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621615:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698931.621618:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621637:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698931.621670:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698931.621688:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698931.621721:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698931.621726:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698931.621727:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698931.621732:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6832 00000100:00080000:0.0:1787698931.621736:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698931.621739:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621742:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621745:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698931.621748:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698931.621752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621754:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621757:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698931.621759:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698931.621768:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787698931.621771:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698931.621780:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698931.621791:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126400256:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787698931.621795:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698931.621808:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698931.621818:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698931.621821:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698931.621823:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698931.621827:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6832 00000100:00080000:0.0:1787698931.621831:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698931.621832:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698931.621839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621846:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621851:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:1.0:1787698931.621852:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621854:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698931.621860:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621865:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698931.621868:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698931.621874:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698931.621879:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698931.621904:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787698931.621913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621920:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698931.621930:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698931.621934:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00000040:1.0:1787698931.621937:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698931.621944:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698931.621952:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6832 00000100:00080000:0.0:1787698931.621957:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698931.621960:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621963:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621972:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698931.621975:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698931.621979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698931/real 1787698931] req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698931.621982:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.621989:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.621994:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698931.622000:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698931.622007:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698931.622011:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698931.622029:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698931.622038:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698931.622047:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698931.622053:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698931.622068:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787698931.622069:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698931.622076:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698931.622078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:0.0:1787698931.622079:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698931.622084:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6832 00000100:00000040:1.0:1787698931.622091:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698931.622098:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787698931.622106:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126400256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698931.622107:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.622114:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.622117:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698931.622120:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698931.622121:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126400256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698931.622125:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698931.622131:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698931.622135:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698931.622142:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698931.622146:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000008:00100000:0.0:1787698931.622148:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698931.622154:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698931.622164:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698931.622171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698931.622179:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698931.622197:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6842) 00000100:00000040:1.0:1787698931.622211:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622219:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698931.622235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698931.622246:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622257:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698931.622264:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698931/real 1787698931] req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698931.622308:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698931.622311:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698931.622318:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698931.622335:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622343:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698931.622360:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622367:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698931.622379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698931.622386:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622395:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698931.622401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698931/real 1787698931] req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622448:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698931.622457:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698931.622477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698931.622482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622496:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698931.622502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622510:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698931.622523:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622531:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698931.622543:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698931.622550:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622560:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698931.622566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698931/real 1787698931] req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622577:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622586:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698931.622593:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698931.622596:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698931.622600:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698931.622614:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126400384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698931.622622:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698931.622627:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126400512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698931.622634:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698931.622639:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126400640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698931.622648:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698931.622656:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698931.622686:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622693:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698931.622704:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698931.622711:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698956 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622720:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698931.622725:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698931/real 1787698931] req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622736:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698931.622744:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698931.622751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698931.622754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698931.622758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698931.622765:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126400768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698931.622770:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126400768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698956 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698931.622777:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698932.133583:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698932.133586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698932.134510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698932.261623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698933.157574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698933.157590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698933.157701:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698933.285600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698933.541590:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301506927, at 4301506944 00000100:00000040:2.0:1787698933.542313:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301506927, at 4301506944 00000100:00100000:3.0:1787698934.181571:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698934.181725:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698934.182023:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698934.309637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698935.205574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698935.205585:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698935.205585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698935.333860:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698936.229510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698936.229510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698936.229540:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698936.357640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698936.741525:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698936.741535:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698936.741538:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698936.741549:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6837 00000100:00080000:0.0:1787698936.741559:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698936.741562:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.741566:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.741570:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698936.741575:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698936.741578:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.741581:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.741584:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698936.741587:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698936.741657:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698936.741683:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698936.741697:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.741772:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.741774:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698936.741782:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698936.741786:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698936.741796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126400896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698936.741823:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.741835:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6837 00000100:00000040:1.0:1787698936.741836:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698936.741843:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698936.741848:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.741851:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.741853:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698936.741856:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698936.741859:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.741862:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.741865:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698936.741868:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000800:00000100:1.0:1787698936.741874:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787698936.741877:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698936.741883:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698936.741892:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698936.741898:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698936.741914:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.741924:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698936.741927:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698936.741931:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787698936.741931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698936/real 1787698936] req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.741939:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6837 00000100:00080000:0.0:1787698936.741945:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698936.741950:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.741953:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.741956:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698936.741959:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00100000:1.0:1787698936.741959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698936.741967:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.741973:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.741978:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:1.0:1787698936.741982:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.741984:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698936.741991:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787698936.741998:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698936.741999:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00080000:1.0:1787698936.742007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787698936.742012:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698936.742016:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698936.742036:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126400896:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698936.742047:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698936.742051:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698936.742055:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698936.742060:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6837 00000100:00000040:1.0:1787698936.742064:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126400896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.742065:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698936.742072:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.742078:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.742082:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698936.742085:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000020:00000040:1.0:1787698936.742085:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787698936.742091:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.742094:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.742096:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698936.742100:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698936.742104:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698936.742108:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698936.742112:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698936.742120:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698936.742127:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787698936.742137:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.742147:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698936.742149:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.742152:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698936.742156:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698936.742164:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6837 00000100:00080000:0.0:1787698936.742169:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698936.742169:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698936.742175:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.742179:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.742181:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698936.742183:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000800:00000100:1.0:1787698936.742188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787698936.742189:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698936.742194:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698936.742196:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698936.742200:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698936.742201:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698936.742210:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698936.742216:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698936.742216:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698936.742227:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698936/real 1787698936] req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698936.742240:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698936.742242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698936.742269:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698936.742291:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6847) 00000100:00080000:1.0:1787698936.742292:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698936.742297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698936.742301:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742309:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698936.742315:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742323:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698936.742341:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742349:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698936.742361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698936.742369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742395:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698936.742402:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698936/real 1787698936] req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742439:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742447:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698936.742455:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698936.742458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698936.742462:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742469:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698936.742474:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126401024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698936.742482:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698936.742487:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126401152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698936.742494:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698936.742501:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742509:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698936.742520:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742526:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698936.742538:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698936.742544:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742552:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698936.742557:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698936/real 1787698936] req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742567:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742574:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698936.742581:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698936.742584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698936.742587:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742594:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698936.742599:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742605:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698936.742615:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742622:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698936.742631:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698936.742637:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698961 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742650:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698936.742655:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698936/real 1787698936] req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698936.742673:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698936.742679:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698936.742682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698936.742686:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698936.742693:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698936.742698:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126401280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698936.742705:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698936.742710:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126401408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698961 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698936.742717:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698937.253533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698937.253537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698937.255708:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698937.381652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698938.277598:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698938.277598:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698938.277610:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698938.405613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698938.661586:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301512047, at 4301512064 00000100:00000040:3.0:1787698938.661587:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301512047, at 4301512064 00000100:00100000:0.0:1787698939.301547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698939.301699:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698939.302583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698939.429648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698940.325538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698940.326008:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698940.326051:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698940.453626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698941.349555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698941.349555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698941.349557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698941.478031:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698941.861521:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698941.861528:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698941.861532:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698941.861539:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6842 00000100:00080000:1.0:1787698941.861548:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698941.861550:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861553:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861555:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698941.861559:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698941.861562:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861564:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861567:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698941.861570:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861583:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698941.861608:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698941.861619:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698941.861637:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698941.861639:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698941.861641:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698941.861643:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6842 00000100:00080000:1.0:1787698941.861646:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698941.861647:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861649:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861651:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698941.861654:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698941.861656:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861659:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861663:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698941.861666:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698941.861675:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698941.861681:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698941.861688:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698941.861693:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698941.861711:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126401536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698941.861713:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698941.861725:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698941.861728:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698941.861730:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698941.861734:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6842 00000100:00080000:1.0:1787698941.861738:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698941.861740:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861746:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698941.861747:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698941.861750:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698941.861755:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698941.861761:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861773:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861777:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861779:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698941.861782:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698941.861788:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698941.861794:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787698941.861797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698941.861807:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698941.861816:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698941.861824:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698941.861833:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698941.861836:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698941.861837:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698941.861840:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6842 00000100:00080000:1.0:1787698941.861843:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698941.861846:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861848:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861849:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698941.861851:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698941.861853:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698941.861857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698941/real 1787698941] req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698941.861858:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861862:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698941.861867:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698941.861874:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698941.861878:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698941.861885:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698941.861886:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698941.861917:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.861920:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698941.861943:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698941.861948:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698941.861950:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698941.861957:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698941.861958:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6842 00000100:00080000:1.0:1787698941.861966:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698941.861966:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698941.861972:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698941.861976:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698941.861977:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861981:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698941.861984:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698941.861989:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698941.861994:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698941.861996:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787698941.861998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126401536:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698941.862000:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698941.862007:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698941.862012:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698941.862013:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126401536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698941.862021:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:2.0:1787698941.862026:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698941.862031:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698941.862039:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6852) 00000100:00000040:2.0:1787698941.862048:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862058:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698941.862073:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862080:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698941.862092:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698941.862099:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862108:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698941.862114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698941/real 1787698941] req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862129:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698941.862144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698941.862147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698941.862151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862157:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698941.862162:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698941.862180:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862187:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698941.862196:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698941.862202:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862210:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698941.862215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698941/real 1787698941] req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862250:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698941.862257:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698941.862260:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698941.862264:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862270:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698941.862275:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862283:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698941.862319:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862326:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698941.862336:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698941.862343:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862351:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698941.862357:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698941/real 1787698941] req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862372:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698941.862386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698941.862389:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698941.862392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862399:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126401920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698941.862411:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862442:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698941.862454:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862461:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698941.862471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698941.862478:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698966 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862486:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698941.862490:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698941/real 1787698941] req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862500:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698941.862507:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698941.862514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698941.862517:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698941.862520:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698941.862527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698941.862533:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126401664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698941.862540:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698941.862546:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126401792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698941.862552:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698941.862557:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126401920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698941.862563:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698941.862567:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126402048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698966 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698941.862574:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698942.373513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698942.373533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698942.374225:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698942.501702:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698943.397554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698943.397555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698943.397726:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698943.525981:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698943.781552:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301517167, at 4301517184 00000100:00000040:3.0:1787698943.781561:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301517167, at 4301517184 00000100:00100000:2.0:1787698944.421568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698944.421569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698944.421788:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698944.550121:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787698945.446374:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698945.446449:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698945.449027:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698945.573624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698946.469597:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698946.469734:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698946.469778:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698946.597802:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698946.981562:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698946.981572:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698946.981575:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698946.981593:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6847 00000100:00080000:1.0:1787698946.981605:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698946.981608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981611:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981614:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698946.981620:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698946.981622:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981623:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981626:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698946.981629:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981647:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698946.981687:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698946.981705:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698946.981730:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698946.981733:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698946.981735:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698946.981738:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6847 00000100:00080000:1.0:1787698946.981741:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698946.981743:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981744:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981746:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698946.981749:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698946.981752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981758:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698946.981761:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698946.981771:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698946.981777:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698946.981783:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698946.981792:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698946.981806:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.981808:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126402176:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698946.981820:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698946.981822:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698946.981824:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698946.981827:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6847 00000100:00080000:1.0:1787698946.981830:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698946.981831:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981833:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981835:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698946.981837:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698946.981839:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981841:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981843:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698946.981848:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698946.981848:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698946.981853:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698946.981860:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698946.981864:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698946.981871:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698946.981878:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698946.981884:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698946.981886:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698946.981887:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698946.981890:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6847 00000100:00080000:1.0:1787698946.981892:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698946.981893:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981895:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981896:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698946.981897:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698946.981899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981900:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981902:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698946.981903:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698946.981907:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698946.981912:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698946.981919:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698946.981923:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981934:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698946.981943:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698946.981946:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698946.981948:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698946.981952:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698946.981960:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6847 00000100:00080000:1.0:1787698946.981964:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698946.981966:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981968:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981970:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698946.981972:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698946.981974:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698946.981975:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698946.981977:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698946.981979:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698946.981983:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698946.981987:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698946.981988:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698946/real 1787698946] req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698946.981997:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698946.982006:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698946.982013:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6858) 00000100:00100000:2.0:1787698946.982019:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698946.982039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698946.982043:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698946.982047:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126402176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698946.982061:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126402176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698946.982071:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698946.982089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698946.982109:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982116:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698946.982126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698946.982133:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982143:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698946.982149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698946/real 1787698946] req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982160:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982168:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698946.982175:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698946.982178:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698946.982182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982189:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698946.982194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982227:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698946.982239:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982247:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698946.982257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698946.982264:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982273:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698946.982279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698946/real 1787698946] req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982289:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698946.982304:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698946.982307:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698946.982311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698946.982323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698946.982352:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982359:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698946.982370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698946.982376:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982385:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698946.982391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698946/real 1787698946] req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698946.982444:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698946.982449:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698946.982453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982461:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698946.982466:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698946.982486:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982498:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698946.982510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698946.982516:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698971 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982524:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698946.982530:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698946/real 1787698946] req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982540:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698946.982549:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698946.982556:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698946.982559:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698946.982563:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698946.982570:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698946.982576:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126402304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698946.982583:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698946.982589:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126402432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698946.982596:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698946.982600:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126402560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698946.982608:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698946.982612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126402688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698971 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698946.982619:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698947.493550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698947.493553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698947.493571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698947.621655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698948.517703:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698948.517704:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698948.517893:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698948.645718:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698948.901543:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301522287, at 4301522304 00000100:00000040:3.0:1787698948.901543:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301522287, at 4301522304 00000100:00100000:0.0:1787698949.541557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698949.541557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698949.541684:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698949.669649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698950.565525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698950.565525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698950.565525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698950.693596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698951.589573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698951.589573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698951.589574:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698951.717597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698952.101518:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698952.101525:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698952.101528:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698952.101539:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6853 00000100:00080000:1.0:1787698952.101549:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698952.101552:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101556:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101560:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698952.101572:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698952.101576:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101578:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101582:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698952.101585:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101598:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698952.101619:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698952.101635:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698952.101657:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698952.101660:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698952.101662:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698952.101667:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6853 00000100:00080000:1.0:1787698952.101670:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698952.101672:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101674:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101675:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698952.101678:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698952.101680:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101682:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101683:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698952.101685:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698952.101692:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698952.101697:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698952.101706:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698952.101713:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698952.101719:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698952.101721:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698952.101723:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698952.101727:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6853 00000100:00080000:1.0:1787698952.101732:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698952.101736:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698952.101736:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698952.101743:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101749:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698952.101755:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698952.101766:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101774:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101776:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698952.101779:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698952.101783:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698952.101789:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698952.101795:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126402816:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787698952.101804:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698952.101816:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698952.101826:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698952.101829:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698952.101831:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698952.101836:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6853 00000100:00080000:1.0:1787698952.101840:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698952.101843:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698952.101848:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101859:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101867:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698952.101870:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698952.101879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698952.101882:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.101895:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698952.101898:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698952.101904:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698952.101909:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698952.101919:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698952.101929:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787698952.101938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787698952.101951:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698952.101955:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698952.101956:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698952.101970:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6853 00000100:00000040:2.0:1787698952.101971:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698952.101975:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698952.101983:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.101996:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.102004:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698952.102008:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698952.102018:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698952.102021:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698952.102022:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698952.102026:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698952.102029:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698952.102033:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698952.102040:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698952.102046:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698952.102059:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6863) 00000100:00100000:2.0:1787698952.102071:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102092:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102102:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.102151:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.102155:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698952.102160:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102169:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126402816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698952.102177:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126402816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698952.102188:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698952.102206:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102216:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698952.102231:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102238:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698952.102251:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698952.102256:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102264:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698952.102268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102280:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.102285:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.102287:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698952.102290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102295:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126402944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698952.102298:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698952.102310:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102315:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698952.102321:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698952.102324:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102330:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698952.102333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102345:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.102349:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.102351:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698952.102353:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102361:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698952.102365:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698952.102377:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102381:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698952.102385:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698952.102388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102393:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698952.102396:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102407:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.102411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.102413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698952.102415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102450:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698952.102454:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698952.102470:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102476:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698952.102486:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698952.102493:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698977 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102502:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698952.102508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698952/real 1787698952] req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102519:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698952.102526:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698952.102531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698952.102533:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698952.102535:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698952.102539:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698952.102543:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126402944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698952.102552:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698952.102556:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126403072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698952.102560:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698952.102563:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126403200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698952.102567:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698952.102570:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126403328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698977 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698952.102573:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698952.613568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698952.613568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698952.613675:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698952.741646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698953.637594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698953.638863:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698953.638940:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698953.765630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698954.021554:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301527407, at 4301527424 00000100:00000040:3.0:1787698954.021554:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301527407, at 4301527424 00000100:00100000:3.0:1787698954.661540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698954.661541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698954.661570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698954.790127:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698955.685517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698955.685517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698955.685629:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698955.814093:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698956.709573:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698956.709573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698956.709611:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698956.837724:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698957.221639:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698957.221649:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698957.221653:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698957.221662:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6858 00000100:00080000:0.0:1787698957.221673:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698957.221677:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221680:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221684:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698957.221688:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698957.221691:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221699:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221702:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698957.221706:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221721:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698957.221745:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698957.221761:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698957.221794:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698957.221798:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698957.221800:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698957.221804:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6858 00000100:00080000:0.0:1787698957.221807:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698957.221810:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221812:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698957.221817:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698957.221820:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221823:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221825:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698957.221827:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698957.221836:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698957.221840:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698957.221851:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698957.221861:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698957.221869:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698957.221872:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698957.221873:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698957.221877:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6858 00000100:00080000:0.0:1787698957.221881:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698957.221884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221888:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698957.221890:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698957.221893:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221895:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221897:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698957.221900:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698957.221904:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698957.221908:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698957.221917:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698957.221926:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698957.221934:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698957.221937:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698957.221939:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698957.221943:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6858 00000100:00080000:0.0:1787698957.221946:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698957.221948:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221951:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221953:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698957.221955:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787698957.221957:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.221960:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.221963:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698957.221965:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698957.221970:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698957.221974:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698957.221982:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698957.221991:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698957.222009:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698957.222013:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698957.222014:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698957.222018:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6858 00000100:00080000:0.0:1787698957.222021:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698957.222023:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.222042:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.222045:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698957.222047:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698957.222049:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698957.222052:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698957.222054:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787698957.222056:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698957.222060:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698957.222068:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698957.222077:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698957.222086:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698957.222096:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6868) 00000100:00000040:2.0:1787698957.222154:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222169:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126403456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698957.222198:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222209:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698957.222244:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698957.222261:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222300:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698957/real 1787698957] req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222318:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698957.222338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698957.222342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698957.222347:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126403456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698957.222378:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698957.222402:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222410:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698957.222471:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698957.222480:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222490:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698957.222496:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698957/real 1787698957] req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698957.222526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698957.222529:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698957.222534:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698957.222547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222555:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698957.222568:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222703:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698957.222718:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698957.222726:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222735:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698957.222741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698957/real 1787698957] req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222756:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222764:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698957.222772:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698957.222775:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698957.222779:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222785:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698957.222796:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222806:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698957.222819:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222826:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698957.222837:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698957.222843:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222852:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698957.222857:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698957/real 1787698957] req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222873:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222880:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698957.222887:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698957.222890:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698957.222893:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698957.222904:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222912:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698957.222923:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222930:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698957.222941:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698957.222948:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698982 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222957:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698957.222962:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698957/real 1787698957] req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.222972:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698957.222982:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698957.222989:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698957.222992:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698957.222995:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698957.223002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126403968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698957.223008:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126403456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698957.223018:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698957.223037:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126403584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698957.223045:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698957.223051:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126403712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698957.223059:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698957.223064:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126403840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698957.223070:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698957.223074:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126403968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698982 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698957.223081:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698957.733520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698957.733533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698957.733577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698957.861653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698958.757541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698958.757541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698958.757541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698958.885628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698959.141543:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301532527, at 4301532544 00000100:00000040:2.0:1787698959.141548:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301532527, at 4301532544 00000100:00100000:0.0:1787698959.781490:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698959.781491:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698959.781500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698959.909614:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698960.805551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698960.805551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698960.805551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698960.933651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698961.829557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698961.829557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698961.829577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698961.957606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698962.341588:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698962.341597:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698962.341601:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698962.341611:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6863 00000100:00080000:3.0:1787698962.341623:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698962.341626:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341629:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341632:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698962.341638:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698962.341641:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341644:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341647:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698962.341650:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341669:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698962.341713:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698962.341729:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698962.341748:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698962.341751:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698962.341773:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698962.341777:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6863 00000100:00080000:3.0:1787698962.341786:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698962.341788:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341792:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698962.341793:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698962.341794:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698962.341797:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698962.341800:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341803:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341805:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698962.341807:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:2.0:1787698962.341809:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126404096:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698962.341821:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698962.341825:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698962.341835:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698962.341841:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698962.341844:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.341850:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698962.341853:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698962.341856:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698962.341858:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698962.341861:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6863 00000100:00080000:3.0:1787698962.341880:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698962.341883:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341885:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341887:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698962.341889:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:2.0:1787698962.341891:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341898:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341900:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341902:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698962.341904:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698962.341909:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698962.341911:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698962.341913:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698962.341932:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698962.341951:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.341958:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698962/real 1787698962] req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698962.341960:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698962.341964:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698962.341965:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698962.341969:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6863 00000100:00080000:3.0:1787698962.341972:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787698962.341975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698962.341975:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.341978:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341980:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698962.341983:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698962.341985:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698962.341986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698962.341988:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.341990:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698962.341993:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:2.0:1787698962.341995:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787698962.341997:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787698962.341999:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:3.0:1787698962.342001:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698962.342003:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698962.342008:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698962.342011:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126404096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698962.342019:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126404096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698962.342022:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698962.342029:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000020:00000040:2.0:1787698962.342030:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:3.0:1787698962.342032:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698962.342034:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698962.342037:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6863 00000100:00080000:3.0:1787698962.342040:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698962.342042:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.342044:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698962.342046:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:2.0:1787698962.342048:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698962.342049:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698962.342052:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698962.342054:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787698962.342056:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404224:192.168.201.136@tcp:38:kworker.0 00000020:00000040:3.0:1787698962.342057:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698962.342059:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698962.342064:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698962.342068:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698962.342070:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787698962.342075:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698962.342078:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698962.342084:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698962.342094:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6873) 00000800:00000100:2.0:1787698962.342098:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698962.342109:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342120:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698962.342126:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698962/real 1787698962] req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342136:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342142:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698962.342147:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698962.342149:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698962.342151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342160:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698962.342163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342168:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698962.342177:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342181:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698962.342188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698962.342192:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342198:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698962.342201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698962/real 1787698962] req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342207:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698962.342216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698962.342218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698962.342220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698962.342227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342232:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698962.342240:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342244:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698962.342248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698962.342252:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342257:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698962.342260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698962/real 1787698962] req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342266:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342271:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698962.342274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698962.342276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698962.342278:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342281:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698962.342285:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126404224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698962.342290:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698962.342293:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126404352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698962.342298:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698962.342301:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126404480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698962.342305:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698962.342309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698962.342319:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342323:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698962.342328:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698962.342332:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698987 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342336:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698962.342339:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698962/real 1787698962] req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698962.342352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698962.342365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698962.342368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698962.342372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698962.342379:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698962.342385:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126404608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698987 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698962.342393:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698962.853559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698962.853559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698962.853560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:1.0:1787698962.981578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698963.877520:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698963.877522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698963.877523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:1.0:1787698964.005651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698964.261511:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301537647, at 4301537664 00000100:00000040:3.0:1787698964.261522:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301537647, at 4301537664 00000100:00100000:1.0:1787698964.901523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698964.901523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698964.901523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698965.029597:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698965.925598:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698965.925599:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698965.925601:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698966.053673:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698966.949517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698966.949517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698966.949517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698967.077658:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698967.461576:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698967.461586:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698967.461590:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698967.461601:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6868 00000100:00080000:1.0:1787698967.461611:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698967.461614:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461617:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461620:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698967.461625:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698967.461628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461630:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461632:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698967.461635:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461649:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698967.461680:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698967.461696:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698967.461722:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698967.461726:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698967.461728:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698967.461732:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6868 00000100:00080000:1.0:1787698967.461735:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698967.461737:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461738:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461740:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698967.461742:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698967.461744:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461746:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461748:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698967.461750:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698967.461759:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698967.461763:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698967.461768:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698967.461774:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698967.461786:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.461789:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126404736:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698967.461802:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698967.461805:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698967.461807:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698967.461812:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6868 00000100:00080000:1.0:1787698967.461815:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698967.461818:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461821:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461823:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698967.461826:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698967.461829:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461831:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461835:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787698967.461837:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698967.461841:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698967.461851:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698967.461867:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461869:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698967.461880:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698967.461886:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698967.461892:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698967.461896:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698967.461897:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698967.461902:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6868 00000100:00080000:1.0:1787698967.461905:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787698967.461907:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461911:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.461916:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461918:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698967.461921:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698967.461924:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698967.461926:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698967.461931:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.461936:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698967.461940:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698967.461946:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698967.461957:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698967.461961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698967/real 1787698967] req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698967.461972:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698967.461978:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698967.461984:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698967.462004:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787698967.462006:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698967.462009:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698967.462013:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698967.462022:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6868 00000100:00080000:2.0:1787698967.462025:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698967.462027:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787698967.462032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698967.462035:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698967.462038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698967.462043:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.462047:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698967.462051:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:2.0:1787698967.462055:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126404736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698967.462057:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698967.462061:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698967.462063:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698967.462067:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698967.462068:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126404736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698967.462072:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698967.462077:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:1.0:1787698967.462078:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698967.462088:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698967.462096:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462099:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698967.462119:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6878) 00000100:00100000:2.0:1787698967.462119:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698967.462135:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462141:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698967.462150:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698967.462155:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462161:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698967.462165:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698967/real 1787698967] req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462178:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698967.462182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698967.462184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698967.462186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462190:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698967.462193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462197:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698967.462204:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462208:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698967.462213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698967.462217:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462223:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698967.462225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698967/real 1787698967] req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698967.462245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698967.462247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698967.462249:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462256:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126404992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698967.462261:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698967.462279:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462286:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698967.462296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698967.462302:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462310:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698967.462315:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698967/real 1787698967] req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698967.462345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698967.462349:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698967.462352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462358:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698967.462364:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126404864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698967.462371:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698967.462377:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126404992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698967.462384:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698967.462388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126405120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698967.462392:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698967.462397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462402:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698967.462413:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462436:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698967.462444:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698967.462449:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698992 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462455:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698967.462458:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698967/real 1787698967] req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462465:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698967.462471:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698967.462475:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698967.462477:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698967.462479:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698967.462482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698967.462485:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126405248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698992 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698967.462490:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698967.973534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698967.973534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698967.973534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698968.101627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698968.997557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698968.997622:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698968.997799:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698969.125650:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698969.381543:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301542767, at 4301542784 00000100:00000040:3.0:1787698969.381543:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301542767, at 4301542784 00000100:00100000:0.0:1787698970.021557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698970.021557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698970.021564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698970.149555:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698971.045526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698971.045526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787698971.045526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698971.173656:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698972.069537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698972.069542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698972.069553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698972.197635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698972.581530:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698972.581539:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698972.581543:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698972.581553:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6873 00000100:00080000:1.0:1787698972.581562:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698972.581565:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581569:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581572:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698972.581578:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698972.581580:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581583:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581585:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698972.581588:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581602:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698972.581624:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698972.581639:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698972.581662:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698972.581665:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698972.581667:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698972.581671:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6873 00000100:00080000:1.0:1787698972.581675:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698972.581677:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581680:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581682:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698972.581684:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698972.581687:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581690:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581693:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698972.581696:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698972.581704:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698972.581709:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698972.581722:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698972.581728:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698972.581743:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126405376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698972.581745:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698972.581758:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698972.581761:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698972.581762:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787698972.581790:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698972.581796:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6873 00000100:00080000:1.0:1787698972.581802:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698972.581807:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581809:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581813:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698972.581818:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698972.581820:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581823:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581826:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698972.581828:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698972.581833:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698972.581838:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787698972.581846:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698972.581854:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698972.581865:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698972.581868:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698972.581891:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698972.581895:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698972.581900:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698972.581905:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6873 00000100:00080000:1.0:1787698972.581909:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698972.581912:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581915:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581919:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:2.0:1787698972.581919:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698972/real 1787698972] req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698972.581922:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698972.581928:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.581934:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.581938:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698972.581942:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698972.581947:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787698972.581950:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698972.581953:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698972.581964:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698972.581969:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698972.581999:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:1.0:1787698972.582001:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698972.582006:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698972.582015:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698972.582017:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698972.582024:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698972.582027:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698972.582034:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6873 00000100:00100000:2.0:1787698972.582036:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126405376:192.168.201.136@tcp:250:kworker.0 00000100:00080000:1.0:1787698972.582038:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698972.582045:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.582051:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698972.582053:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698972.582057:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698972.582057:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126405376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698972.582064:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698972.582070:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698972.582076:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:1.0:1787698972.582079:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698972.582083:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698972.582087:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698972.582091:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698972.582100:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698972.582110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698972.582112:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698972.582142:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6883) 00000100:00100000:2.0:1787698972.582144:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698972.582162:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582171:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698972.582183:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698972.582190:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582199:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698972.582205:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698972/real 1787698972] req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698972.582234:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698972.582237:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698972.582240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582254:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698972.582259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698972.582277:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582293:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698972.582309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698972.582315:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582323:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698972.582329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698972/real 1787698972] req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582348:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698972.582355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698972.582357:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698972.582360:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582367:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698972.582372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698972.582391:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582398:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698972.582407:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698972.582413:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582446:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698972.582452:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698972/real 1787698972] req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582463:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698972.582477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698972.582479:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698972.582482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582489:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698972.582494:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126405504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698972.582502:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698972.582508:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126405632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698972.582515:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698972.582519:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126405760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698972.582525:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698972.582532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582539:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698972.582549:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582556:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698972.582566:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698972.582573:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787698997 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582581:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698972.582586:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698972/real 1787698972] req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698972.582604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698972.582616:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698972.582619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698972.582622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698972.582629:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126405888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698972.582633:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126405888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787698997 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698972.582639:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698973.093480:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698973.093526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698973.093569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698973.221640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698974.117547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698974.117548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698974.117553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698974.245655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698974.501512:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301547887, at 4301547904 00000100:00000040:3.0:1787698974.501512:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301547887, at 4301547904 00000100:00100000:0.0:1787698975.141533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698975.141533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698975.141542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698975.269606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698976.165550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698976.165550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787698976.165608:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698976.293643:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698977.189539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698977.189539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698977.189562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698977.317652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787698977.701531:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698977.701544:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787698977.701549:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698977.701561:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6878 00000100:00080000:2.0:1787698977.701574:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698977.701578:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701583:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701587:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787698977.701592:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787698977.701596:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701598:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701602:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787698977.701606:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701623:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698977.701656:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698977.701675:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698977.701716:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698977.701721:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787698977.701723:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698977.701727:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6878 00000100:00080000:2.0:1787698977.701731:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698977.701734:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701737:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701739:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787698977.701749:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787698977.701753:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701758:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787698977.701761:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698977.701771:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698977.701777:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698977.701787:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698977.701797:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698977.701816:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698977.701820:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787698977.701822:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698977.701826:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6878 00000100:00080000:2.0:1787698977.701830:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698977.701833:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701835:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701837:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787698977.701840:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787698977.701843:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701845:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701847:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787698977.701850:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698977.701855:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787698977.701859:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787698977.701868:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698977.701877:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698977.701886:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698977.701889:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787698977.701891:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698977.701895:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6878 00000100:00080000:2.0:1787698977.701899:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698977.701902:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701905:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701907:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787698977.701909:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787698977.701912:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701917:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787698977.701920:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698977.701925:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698977.701932:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698977.701940:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698977.701950:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698977.701958:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787698977.701961:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787698977.701963:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787698977.701967:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6878 00000100:00080000:2.0:1787698977.701970:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698977.701973:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701976:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701978:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787698977.701980:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787698977.701983:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787698977.701986:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787698977.701988:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787698977.701991:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787698977.701996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787698977.702000:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787698977.702008:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787698977.702018:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698977.702029:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6888) 00000100:00000040:1.0:1787698977.702099:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702115:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126406016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698977.702206:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702216:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698977.702262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698977.702282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698977/real 1787698977] req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702355:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698977.702365:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698977.702370:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698977.702374:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702382:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126406016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698977.702390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702399:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698977.702415:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702443:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698977.702456:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698977.702464:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702475:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698977.702482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698977/real 1787698977] req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702493:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702502:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698977.702510:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698977.702514:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698977.702518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702526:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698977.702532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698977.702553:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702560:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698977.702571:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698977.702578:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702586:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698977.702590:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698977/real 1787698977] req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702610:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698977.702617:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698977.702620:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698977.702624:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698977.702637:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702646:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698977.702659:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702667:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698977.702688:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698977.702697:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702707:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698977.702712:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698977/real 1787698977] req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702724:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702733:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698977.702741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698977.702744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698977.702749:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702757:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698977.702762:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698977.702784:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702791:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698977.702803:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698977.702810:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699002 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702819:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698977.702825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698977/real 1787698977] req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702836:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698977.702845:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698977.702851:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698977.702855:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698977.702858:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698977.702866:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698977.702872:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126406016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698977.702884:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698977.702893:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126406144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698977.702900:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698977.702906:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126406272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698977.702912:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698977.702918:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126406400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698977.702925:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698977.702930:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126406528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699002 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698977.702937:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698978.213531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698978.213531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698978.213531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698978.341651:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698979.237493:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698979.237493:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698979.237498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698979.365636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698979.621544:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301553007, at 4301553024 00000100:00000040:3.0:1787698979.621544:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301553007, at 4301553024 00000100:00100000:1.0:1787698980.261588:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698980.261588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698980.261734:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698980.389598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698981.285544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698981.285544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698981.285544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698981.413610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698982.309518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698982.309518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698982.309518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698982.437573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698982.821582:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698982.821591:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698982.821618:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698982.821629:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6883 00000100:00080000:3.0:1787698982.821641:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698982.821643:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698982.821647:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698982.821650:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698982.821654:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698982.821656:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698982.821658:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698982.821660:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698982.821663:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698982.821679:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698982.821704:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698982.821716:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698982.821735:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698982.821738:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698982.821756:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698982.821760:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6883 00000100:00080000:3.0:1787698982.821764:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698982.821766:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698982.821768:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698982.821770:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698982.821772:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787698982.821776:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698982.821779:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698982.821779:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698982.821782:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698982.821786:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:1.0:1787698982.821799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126406656:192.168.201.136@tcp:250:kworker.0 00000008:00100000:3.0:1787698982.821808:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698982.821814:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698982.821825:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698982.821828:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.821847:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698982.821851:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698982.821863:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698982.821866:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698982.821868:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698982.821872:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6883 00000100:00080000:3.0:1787698982.821877:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698982.821879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698982.821882:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698982.821885:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000800:00000100:1.0:1787698982.821886:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698982.821888:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698982.821899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698982.821902:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698982.821907:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698982.821907:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698982.821914:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698982.821922:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698982.821928:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698982.821942:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698982.821945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698982/real 1787698982] req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698982.821964:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.821973:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698982.821976:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698982.821986:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698982.821989:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698982.821990:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698982.821999:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6883 00000100:00080000:3.0:1787698982.822007:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787698982.822009:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:3.0:1787698982.822012:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:1.0:1787698982.822017:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787698982.822019:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698982.822026:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698982.822033:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698982.822038:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:1.0:1787698982.822041:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126406656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787698982.822045:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698982.822051:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698982.822055:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126406656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698982.822055:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698982.822061:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698982.822070:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698982.822075:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:3.0:1787698982.822077:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698982.822100:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698982.822109:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698982.822129:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822131:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406784:192.168.201.136@tcp:38:kworker.0 00000100:00080000:3.0:1787698982.822146:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698982.822149:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698982.822153:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698982.822157:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698982.822158:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6883 00000100:00080000:3.0:1787698982.822166:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698982.822173:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698982.822176:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698982.822180:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698982.822183:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698982.822186:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787698982.822189:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:1.0:1787698982.822195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698982.822195:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698982.822200:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698982.822204:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698982.822208:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787698982.822212:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787698982.822220:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698982.822225:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787698982.822234:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698982.822236:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698982/real 1787698982] req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698982.822261:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822265:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698982.822282:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6893) 00000100:00000040:1.0:1787698982.822293:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698982.822302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698982.822306:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698982.822310:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822319:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698982.822325:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822334:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698982.822346:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822352:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698982.822361:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698982.822366:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822372:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698982.822376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698982/real 1787698982] req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698982.822401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698982.822404:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698982.822408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126406912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698982.822447:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126406784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698982.822456:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698982.822462:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126406912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698982.822469:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698982.822477:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822483:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698982.822497:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822502:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698982.822510:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698982.822515:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822521:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698982.822524:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698982/real 1787698982] req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822539:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698982.822554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698982.822557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698982.822561:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822568:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698982.822574:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822580:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698982.822588:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822605:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698982.822617:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698982.822624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699007 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822633:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698982.822638:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698982/real 1787698982] req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822649:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698982.822658:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698982.822664:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698982.822666:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698982.822668:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698982.822673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698982.822676:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126407040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698982.822683:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698982.822689:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126407168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699007 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698982.822695:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698983.333549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698983.333549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698983.333696:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698983.461601:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698984.357551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698984.357551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698984.357551:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698984.485657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698984.741592:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301558127, at 4301558144 00000100:00000040:3.0:1787698984.742435:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301558127, at 4301558145 00000100:00100000:1.0:1787698985.381562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698985.381569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698985.381572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698985.509707:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698986.405565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698986.405565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698986.405720:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698986.533710:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698987.429557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698987.429557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698987.429559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698987.557625:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787698987.941546:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698987.941559:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787698987.941564:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698987.941578:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6888 00000100:00080000:3.0:1787698987.941591:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698987.941595:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.941600:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941604:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787698987.941609:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787698987.941614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.941617:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941620:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787698987.941624:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941641:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787698987.941674:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698987.941691:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698987.941715:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698987.941720:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787698987.941722:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698987.941726:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6888 00000100:00080000:3.0:1787698987.941730:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698987.941733:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.941736:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941756:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787698987.941759:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698987.941763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698987.941765:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.941773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941778:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787698987.941784:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698987.941795:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698987.941800:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698987.941803:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126407296:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787698987.941820:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698987.941832:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698987.941842:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:1.0:1787698987.941845:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698987.941847:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787698987.941850:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698987.941859:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6888 00000100:00080000:3.0:1787698987.941865:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698987.941871:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.941877:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698987.941882:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941883:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787698987.941888:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787698987.941892:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.941895:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941897:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787698987.941900:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698987.941905:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787698987.941916:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787698987.941927:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787698987.941937:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698987.941945:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698987.941949:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787698987.941951:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787698987.941956:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6888 00000100:00080000:3.0:1787698987.941961:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698987.941964:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.941967:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941970:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787698987.941974:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787698987.941979:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698987.941982:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698987.941982:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.941987:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787698987.941993:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787698987.942002:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1787698987.942005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407424:192.168.201.136@tcp:38:kworker.0 00000020:00000040:3.0:1787698987.942009:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787698987.942023:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698987.942027:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698987.942042:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698987.942065:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787698987.942067:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787698987.942077:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787698987.942079:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:1.0:1787698987.942080:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787698987.942085:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6888 00000100:00000040:1.0:1787698987.942089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787698987.942092:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787698987.942100:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698987.942103:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787698987.942107:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787698987.942112:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787698987.942115:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698987.942119:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787698987.942121:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787698987.942129:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698987.942134:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787698987.942134:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787698987.942140:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698987.942143:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000008:00100000:3.0:1787698987.942147:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:1.0:1787698987.942153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787698987.942154:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698987.942165:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407680:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787698987.942171:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698987.942177:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698987.942195:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787698987.942198:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698987.942203:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787698987.942224:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6899) 00000100:00000040:1.0:1787698987.942236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698987.942252:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698987.942264:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698987.942271:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698987.942281:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698988.453586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698988.453586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787698988.455582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698988.581713:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698989.477572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698989.477572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698989.477572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698989.605708:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787698989.861551:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301563247, at 4301563264 00000100:00000040:2.0:1787698989.861552:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301563247, at 4301563264 00000100:00100000:1.0:1787698990.501553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698990.501554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698990.501643:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698990.629668:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787698991.013904:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787698991.013970:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787698991.013991:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698991.014040:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698991.014085:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698991.014096:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698991.014105:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698991.014114:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698991.014124:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698991.014132:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787698991.014141:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787698991.014150:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699012 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698987/real 1787698991] req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014360:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698991.014370:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698991.014382:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698991.014387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698991.014392:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014401:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407808:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698991.014408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698987/real 1787698991] req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014448:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698991.014470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698991.014477:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698991.014481:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698991.014484:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014491:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407680:192.168.201.136@tcp:8:kworker.0 00000100:00100000:1.0:1787698991.014497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698987/real 1787698991] req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698991.014516:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698991.014524:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698991.014527:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698991.014531:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407424:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698991.014544:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698987/real 1787698991] req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014560:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698991.014567:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698991.014575:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698991.014578:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698991.014582:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014588:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126407552:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787698991.014594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698987/real 1787698991] req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698991.014615:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698991.014623:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698991.014627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698991.014631:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698991.014639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126407296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698991.014647:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126407808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698991.014663:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698991.014675:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126407680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698991.014684:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698991.014689:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126407424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698991.014695:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698991.014700:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126407552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698991.014705:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698991.014710:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126407296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699012 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698991.014717:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787698991.525568:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698991.525568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698991.525569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698991.653689:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698992.549497:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698992.549497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698992.549500:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698992.677648:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787698993.061568:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698993.061581:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787698993.061585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698993.061598:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6894 00000100:00080000:0.0:1787698993.061612:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698993.061617:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.061621:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061625:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787698993.061631:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787698993.061634:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.061637:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061641:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787698993.061644:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061658:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787698993.061690:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698993.061707:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698993.061734:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698993.061738:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787698993.061740:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698993.061745:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6894 00000100:00080000:0.0:1787698993.061749:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698993.061752:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.061755:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061757:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787698993.061760:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787698993.061763:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.061766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061768:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787698993.061771:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698993.061782:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698993.061788:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698993.061798:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698993.061802:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698993.061826:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.061830:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126407936:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787698993.061841:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698993.061846:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787698993.061847:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698993.061852:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6894 00000100:00080000:0.0:1787698993.061856:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698993.061859:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.061862:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061865:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787698993.061867:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787698993.061873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698993.061873:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698993.061879:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061885:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787698993.061890:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:1.0:1787698993.061895:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787698993.061899:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787698993.061905:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787698993.061917:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698993.061927:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698993.061936:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787698993.061940:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787698993.061943:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787698993.061945:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787698993.061950:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6894 00000100:00080000:0.0:1787698993.061956:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698993.061959:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.061962:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061965:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787698993.061968:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698993.061969:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698993.061975:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.061982:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.061987:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787698993.061990:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698993.061996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787698993.062002:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698993.062013:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787698993.062024:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698993.062032:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787698993.062034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698993.062038:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787698993.062042:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787698993.062051:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6894 00000100:00080000:0.0:1787698993.062057:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787698993.062061:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787698993.062064:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787698993.062066:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.062072:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787698993.062076:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787698993.062080:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787698993.062084:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787698993.062089:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:1.0:1787698993.062093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787698993.062094:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787698993.062102:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1787698993.062108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787698993.062109:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787698993.062117:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698993.062124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787698993.062129:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787698993.062142:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126407936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787698993.062149:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062154:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126407936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698993.062172:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787698993.062191:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062199:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698993.062213:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062221:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698993.062233:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698993.062247:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062257:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787698993.062263:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062275:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698993.062290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698993.062293:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698993.062297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062304:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698993.062309:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698993.062329:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062337:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698993.062348:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698993.062354:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062363:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787698993.062368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062385:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062395:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698993.062403:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698993.062406:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698993.062410:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062458:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787698993.062465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062474:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698993.062488:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062495:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698993.062508:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698993.062514:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062530:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787698993.062536:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062554:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062562:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698993.062574:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698993.062577:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698993.062581:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698993.062596:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126408064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698993.062605:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787698993.062611:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126408192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698993.062621:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787698993.062626:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6904) 00000100:00000040:1.0:1787698993.062628:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126408320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698993.062636:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787698993.062644:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062652:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698993.062669:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062675:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787698993.062687:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698993.062695:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699018 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062703:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787698993.062708:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698993/real 1787698993] req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062718:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698993.062726:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698993.062733:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787698993.062737:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787698993.062740:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787698993.062747:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787698993.062752:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126408448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699018 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698993.062759:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698993.573559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698993.573565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698993.573570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698993.701622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787698994.597518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698994.597519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698994.597519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698994.725628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787698994.981562:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301568367, at 4301568384 00000100:00000040:3.0:1787698994.981562:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301568367, at 4301568384 00000100:00100000:1.0:1787698995.621562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787698995.621562:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698995.621704:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787698995.749594:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698996.645572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698996.645572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787698996.645792:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698996.773641:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787698997.669555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698997.669555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787698997.669577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698997.797606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787698998.181561:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698998.181579:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787698998.181585:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698998.181597:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6899 00000100:00080000:1.0:1787698998.181610:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698998.181612:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181616:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181619:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787698998.181625:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787698998.181627:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181629:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787698998.181633:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181648:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787698998.181673:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698998.181685:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698998.181706:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698998.181709:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787698998.181710:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698998.181713:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6899 00000100:00080000:1.0:1787698998.181716:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698998.181717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181722:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787698998.181724:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787698998.181726:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181728:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181730:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787698998.181731:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698998.181739:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787698998.181741:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787698998.181743:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787698998.181755:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787698998.181762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126408576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787698998.181763:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698998.181773:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698998.181776:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787698998.181776:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698998.181778:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6899 00000100:00080000:1.0:1787698998.181780:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698998.181781:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181784:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787698998.181785:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787698998.181787:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181789:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181796:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787698998.181798:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787698998.181802:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787698998.181806:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787698998.181812:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698998.181820:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787698998.181821:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698998.181828:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698998.181834:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698998.181835:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787698998.181836:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698998.181840:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6899 00000100:00080000:1.0:1787698998.181842:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787698998.181844:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181845:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181846:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787698998.181848:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787698998.181850:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181851:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181852:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787698998.181853:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698998.181856:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698998.181861:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698998.181866:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787698998.181871:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181877:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787698998.181887:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787698998.181889:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787698998.181890:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787698998.181892:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6899 00000100:00080000:1.0:1787698998.181894:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787698998.181897:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698998.181899:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181904:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181908:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787698998.181911:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787698998.181915:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787698998.181917:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787698998.181918:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787698998.181924:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787698998.181928:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787698998.181931:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787698998.181937:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787698998.181949:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.181949:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698998/real 1787698998] req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787698998.181981:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6909) 00000100:00100000:2.0:1787698998.181986:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.181995:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698998.182006:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698998.182010:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698998.182014:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182023:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126408576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787698998.182030:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126408576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698998.182040:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787698998.182064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182074:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698998.182089:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182097:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698998.182109:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698998.182117:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182127:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787698998.182133:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698998/real 1787698998] req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182143:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182150:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698998.182158:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698998.182161:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698998.182165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698998.182177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182192:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698998.182204:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182211:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698998.182222:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698998.182229:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182237:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787698998.182248:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698998/real 1787698998] req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182268:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698998.182276:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698998.182279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698998.182283:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787698998.182296:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182304:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698998.182318:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182325:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698998.182352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698998.182360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182369:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787698998.182375:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698998/real 1787698998] req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182395:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698998.182402:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698998.182405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698998.182409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182413:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126408960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698998.182439:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182449:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698998.182462:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182473:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787698998.182485:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787698998.182491:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699023 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182499:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787698998.182504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787698998/real 1787698998] req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787698998.182522:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787698998.182528:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787698998.182531:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787698998.182534:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787698998.182542:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787698998.182548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126408704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698998.182556:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698998.182563:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126408832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698998.182571:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787698998.182575:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126408960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698998.182582:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787698998.182587:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126409088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699023 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787698998.182593:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787698998.693544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698998.693544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698998.693544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698998.821650:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787698999.717563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787698999.717563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787698999.717563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787698999.845652:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699000.101536:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301573487, at 4301573504 00000100:00000040:3.0:1787699000.101536:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301573487, at 4301573504 00000100:00100000:0.0:1787699000.741564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699000.741578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699000.741692:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699000.869649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699001.765542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699001.765542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699001.765542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699001.893649:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699002.789550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699002.789550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699002.789550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699002.917621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699003.301579:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699003.301617:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699003.301621:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699003.301634:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6904 00000100:00080000:3.0:1787699003.301647:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699003.301651:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.301655:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.301659:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699003.301665:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699003.301668:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.301671:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.301674:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699003.301678:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699003.301693:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699003.301724:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699003.301741:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699003.301767:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699003.301770:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699003.301772:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699003.301777:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6904 00000100:00080000:3.0:1787699003.301781:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699003.301783:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.301786:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.301789:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699003.301793:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699003.301804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.301808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.301811:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:2.0:1787699003.301817:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699003.301817:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699003.301832:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699003.301839:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126409216:192.168.201.136@tcp:250:kworker.0 00000020:00000040:3.0:1787699003.301840:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699003.301857:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699003.301868:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.301878:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699003.301880:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699003.301891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699003.301895:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787699003.301901:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699003.301905:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6904 00000100:00080000:3.0:1787699003.301911:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699003.301914:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.301918:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.301920:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699003.301923:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699003.301926:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.301929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.301932:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699003.301935:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699003.301940:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699003.301945:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787699003.301948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787699003.301959:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699003.301972:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699003.302008:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699003.302019:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699003.302023:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699003.302025:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699003.302030:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6904 00000100:00080000:3.0:1787699003.302036:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787699003.302040:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699003/real 1787699003] req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699003.302042:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.302050:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.302055:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699003.302060:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699003.302064:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.302070:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787699003.302071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699003.302075:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699003.302082:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699003.302089:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699003.302096:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699003.302104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699003.302111:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699003.302118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699003.302128:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699003.302130:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699003.302144:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699003.302155:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699003.302158:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126409216:192.168.201.136@tcp:250:kworker.0 00000100:00080000:3.0:1787699003.302159:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787699003.302170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126409216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699003.302176:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6904 00000100:00080000:3.0:1787699003.302185:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000020:00000040:2.0:1787699003.302191:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699003.302191:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.302197:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.302201:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699003.302204:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699003.302207:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699003.302210:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699003.302212:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:2.0:1787699003.302219:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699003.302220:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699003.302230:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699003.302236:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409344:192.168.201.136@tcp:38:kworker.0 00000020:00000040:3.0:1787699003.302237:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699003.302270:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699003.302273:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699003.302287:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699003.302302:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787699003.302303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699003.302314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699003.302320:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6914) 00000100:00000040:2.0:1787699003.302329:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787699003.302337:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699003/real 1787699003] req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302350:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302361:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699003.302370:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699003.302374:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699003.302378:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302385:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699003.302391:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302400:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699003.302412:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302461:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699003.302472:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699003.302480:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302489:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787699003.302503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699003/real 1787699003] req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302523:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699003.302531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699003.302534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699003.302538:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699003.302552:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699003.302574:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302589:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699003.302600:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699003.302606:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302616:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787699003.302622:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699003/real 1787699003] req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302632:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302640:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699003.302647:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699003.302650:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699003.302654:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699003.302667:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126409344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699003.302674:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699003.302681:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126409472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699003.302697:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699003.302703:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126409600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699003.302711:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699003.302719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302727:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699003.302739:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302747:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699003.302758:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699003.302765:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699028 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302774:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787699003.302780:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699003/real 1787699003] req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302790:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699003.302798:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699003.302806:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699003.302809:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699003.302813:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699003.302819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699003.302822:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126409728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699028 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699003.302827:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699003.813527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699003.813528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699003.813633:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699003.941639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699004.838250:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699004.838274:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699004.838584:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699004.965622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699005.221583:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301578607, at 4301578624 00000100:00000040:3.0:1787699005.221584:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301578607, at 4301578624 00000100:00100000:1.0:1787699005.861527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699005.861527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699005.861554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699005.989618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699006.885510:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699006.885510:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699006.885510:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699007.013593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699007.909547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699007.909547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699007.909547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699008.038111:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699008.421556:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699008.421569:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699008.421575:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699008.421590:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6909 00000100:00080000:0.0:1787699008.421605:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699008.421609:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421613:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421616:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699008.421621:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699008.421624:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421627:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699008.421635:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421653:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699008.421687:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699008.421702:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699008.421719:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699008.421722:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699008.421723:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699008.421726:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6909 00000100:00080000:0.0:1787699008.421730:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699008.421733:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421737:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699008.421739:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699008.421742:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421743:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421745:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699008.421747:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699008.421757:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787699008.421757:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699008.421762:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699008.421774:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699008.421781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126409856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699008.421785:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699008.421798:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699008.421800:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699008.421801:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699008.421804:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6909 00000100:00080000:0.0:1787699008.421807:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699008.421809:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421810:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421812:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699008.421813:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699008.421815:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421817:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421819:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:2.0:1787699008.421823:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699008.421824:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699008.421830:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699008.421835:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699008.421843:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787699008.421844:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699008.421852:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699008.421860:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699008.421863:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699008.421865:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699008.421869:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6909 00000100:00080000:0.0:1787699008.421872:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699008.421875:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421878:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421881:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699008.421883:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699008.421886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421889:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.421892:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000800:00000100:2.0:1787699008.421894:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699008.421897:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699008.421904:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699008.421909:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699008.421924:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699008.421924:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699008.421950:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699008.421964:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699008.421968:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699008/real 1787699008] req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699008.421969:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699008.421973:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699008.421980:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6909 00000100:00080000:0.0:1787699008.421987:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699008.421993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.421997:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699008.422000:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699008.422003:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:2.0:1787699008.422007:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699008.422008:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699008.422022:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699008.422024:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699008.422026:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699008.422033:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699008.422042:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1787699008.422043:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:0.0:1787699008.422050:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787699008.422051:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699008.422059:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699008.422065:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699008.422071:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126409856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699008.422084:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422086:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126409856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699008.422101:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6919) 00000020:00000040:2.0:1787699008.422115:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787699008.422131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422140:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699008.422155:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422175:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699008.422188:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699008.422195:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422206:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787699008.422212:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699008/real 1787699008] req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422232:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699008.422248:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699008.422251:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699008.422255:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422263:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126409984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699008.422269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422277:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699008.422289:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422297:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699008.422307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699008.422314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422323:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787699008.422327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699008/real 1787699008] req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422351:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699008.422359:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699008.422362:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699008.422366:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422374:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699008.422380:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422387:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699008.422400:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422407:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699008.422416:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699008.422445:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422455:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787699008.422461:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699008/real 1787699008] req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422476:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422484:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699008.422491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699008.422495:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699008.422499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422505:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699008.422510:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422516:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699008.422528:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422535:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699008.422545:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699008.422552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699033 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422561:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787699008.422566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699008/real 1787699008] req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422576:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699008.422584:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699008.422591:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699008.422594:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699008.422598:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699008.422605:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699008.422610:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126409984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699008.422618:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699008.422624:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126410112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699008.422631:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699008.422637:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126410240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699008.422643:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699008.422648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126410368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699033 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699008.422655:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699008.933560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699008.933560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699008.933669:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699009.061817:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699009.957535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699009.957535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699009.957650:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699010.085626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699010.341546:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301583727, at 4301583744 00000100:00000040:3.0:1787699010.341546:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301583727, at 4301583744 00000100:00100000:1.0:1787699010.981539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699010.981539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699010.981539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699011.109621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699012.005521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699012.005521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699012.005525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699012.133622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699013.029544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699013.029544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699013.029544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699013.157637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699013.541710:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699013.541725:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699013.541729:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699013.541746:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6914 00000100:00080000:1.0:1787699013.541759:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699013.541763:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.541768:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.541771:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699013.541778:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699013.541782:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.541784:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.541788:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699013.541791:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699013.541810:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699013.541847:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699013.541866:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699013.541893:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699013.541898:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699013.541900:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699013.541905:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6914 00000100:00080000:1.0:1787699013.541909:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699013.541912:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.541915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.541918:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699013.541921:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699013.541924:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.541927:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.541930:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699013.541933:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699013.541945:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:2.0:1787699013.541948:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699013.541952:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699013.541970:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699013.541973:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126410496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787699013.541994:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699013.542004:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699013.542008:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699013.542010:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699013.542014:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6914 00000100:00080000:1.0:1787699013.542019:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699013.542019:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699013.542026:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.542033:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.542037:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000100:00000040:2.0:1787699013.542038:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787699013.542042:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699013.542048:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.542050:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.542053:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787699013.542055:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699013.542061:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699013.542067:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699013.542086:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699013.542097:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699013.542107:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787699013.542107:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699013.542113:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699013.542116:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699013.542122:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6914 00000100:00080000:1.0:1787699013.542126:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699013.542129:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.542132:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.542136:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:2.0:1787699013.542136:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699013.542141:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699013.542148:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699013.542152:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.542155:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699013.542158:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699013.542164:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699013.542169:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699013.542179:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699013.542187:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699013/real 1787699013] req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699013.542193:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699013.542208:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699013.542212:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699013.542214:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787699013.542217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699013.542220:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6914 00000100:00080000:1.0:1787699013.542228:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699013.542233:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699013.542237:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699013.542239:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699013.542245:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699013.542249:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699013.542255:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787699013.542256:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:1.0:1787699013.542261:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:2.0:1787699013.542265:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787699013.542266:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699013.542273:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787699013.542273:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699013.542281:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699013.542289:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699013.542290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126410496:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787699013.542300:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699013.542303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126410496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699013.542328:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699013.542336:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787699013.542341:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6924) 00000100:00000040:2.0:1787699013.542357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542368:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699013.542385:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542393:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699013.542443:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699013.542452:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542464:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787699013.542470:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699013/real 1787699013] req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542481:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542489:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699013.542497:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699013.542500:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699013.542504:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542512:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699013.542519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542541:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699013.542555:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542563:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699013.542574:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699013.542581:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542590:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787699013.542596:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699013/real 1787699013] req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542607:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542616:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699013.542624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699013.542627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699013.542632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542639:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699013.542644:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542654:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699013.542666:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542673:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699013.542683:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699013.542689:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542698:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787699013.542703:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699013/real 1787699013] req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542723:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699013.542741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699013.542744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699013.542748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542755:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126410880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699013.542761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542769:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699013.542780:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542788:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699013.542797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699013.542803:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699038 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542811:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787699013.542817:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699013/real 1787699013] req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542828:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699013.542836:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699013.542844:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699013.542847:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699013.542851:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699013.542858:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699013.542864:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126410624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699013.542873:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699013.542879:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126410752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699013.542892:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699013.542898:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126410880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699013.542905:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699013.542910:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126411008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699038 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699013.542918:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699014.053531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699014.053531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699014.054696:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699014.181658:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699015.077550:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699015.077550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699015.077550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699015.205629:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699015.461552:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301588847, at 4301588864 00000100:00000040:2.0:1787699015.461553:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301588847, at 4301588864 00000100:00100000:2.0:1787699016.101519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699016.101526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699016.101526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699016.229622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699017.125561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699017.125567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699017.125700:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699017.253620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699018.149541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699018.149541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699018.149544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699018.277650:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699018.661570:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699018.661584:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699018.661589:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699018.661601:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6919 00000100:00080000:0.0:1787699018.661614:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699018.661618:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699018.661622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.661626:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699018.661631:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699018.661635:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699018.661637:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.661640:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699018.661652:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699018.661671:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699018.661711:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699018.661726:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699018.661750:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699018.661754:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699018.661757:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699018.661762:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6919 00000100:00080000:0.0:1787699018.661768:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699018.661771:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699018.661775:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.661778:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699018.661781:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699018.661789:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699018.661792:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699018.661793:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.661800:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699018.661811:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699018.661832:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787699018.661838:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126411136:192.168.201.136@tcp:250:kworker.0 00000020:00000040:0.0:1787699018.661843:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699018.661864:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699018.661878:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.661888:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.661913:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699018.661915:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699018.661921:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699018.661923:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699018.661930:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6919 00000100:00080000:0.0:1787699018.661935:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699018.661939:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699018.661942:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.661945:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699018.661948:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:2.0:1787699018.661953:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699018.661961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699018.661967:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.661969:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699018.661976:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:2.0:1787699018.661978:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699018.661996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699018.662008:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699018.662022:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699018.662033:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699018/real 1787699018] req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699018.662040:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699018.662073:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787699018.662076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699018.662079:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699018.662093:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:2.0:1787699018.662094:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699018.662116:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6919 00000100:00080000:2.0:1787699018.662118:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699018.662122:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699018.662131:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:2.0:1787699018.662133:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699018.662138:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.662146:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:2.0:1787699018.662146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699018.662153:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699018.662169:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:2.0:1787699018.662174:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126411136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699018.662180:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.662186:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:2.0:1787699018.662188:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126411136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699018.662195:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699018.662216:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699018.662219:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:0.0:1787699018.662227:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699018.662241:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699018.662243:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787699018.662284:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699018.662291:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662305:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699018.662313:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:2.0:1787699018.662315:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699018.662323:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699018.662325:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699018.662334:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6919 00000800:00000100:2.0:1787699018.662339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699018.662344:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699018.662350:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699018.662356:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699018.662359:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.662367:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699018.662373:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699018.662382:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699018.662385:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:0.0:1787699018.662388:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699018.662394:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:2.0:1787699018.662397:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699018/real 1787699018] req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699018.662400:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699018.662413:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699018.662455:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699018.662472:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:2.0:1787699018.662485:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787699018.662492:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699018.662492:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699018.662499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662546:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699018.662547:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662574:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699018.662578:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6929) 00000100:00100000:2.0:1787699018.662591:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699018.662609:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662616:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699018.662629:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699018.662635:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662643:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787699018.662649:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699018/real 1787699018] req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662665:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662673:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699018.662680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699018.662684:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699018.662688:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699018.662702:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126411264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699018.662708:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699018.662713:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126411392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699018.662717:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699018.662723:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662729:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699018.662739:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662743:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699018.662751:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699018.662755:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662761:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787699018.662764:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699018/real 1787699018] req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662775:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699018.662780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699018.662782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699018.662784:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662787:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699018.662791:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662796:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699018.662802:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662806:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699018.662811:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699018.662815:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699043 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662823:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787699018.662826:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699018/real 1787699018] req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662832:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699018.662837:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699018.662840:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699018.662842:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699018.662844:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699018.662848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699018.662850:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126411520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699018.662855:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699018.662858:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126411648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699043 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699018.662862:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699019.173538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699019.173538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699019.173538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699019.301604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699020.197532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699020.197532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699020.197532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699020.325580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699020.581534:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301593967, at 4301593984 00000100:00000040:3.0:1787699020.581534:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301593967, at 4301593984 00000100:00100000:1.0:1787699021.221504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699021.221505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699021.221505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699021.349610:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699022.245518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699022.245518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699022.245518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699022.373607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699023.269503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699023.269504:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699023.269588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699023.397598:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699023.781527:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699023.781535:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699023.781538:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699023.781545:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6924 00000100:00080000:0.0:1787699023.781554:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699023.781556:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781559:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781562:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699023.781565:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699023.781567:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781569:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781571:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699023.781573:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781584:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699023.781603:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699023.781613:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699023.781633:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699023.781635:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699023.781636:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699023.781639:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6924 00000100:00080000:0.0:1787699023.781641:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699023.781642:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781644:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781646:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699023.781648:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699023.781649:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781651:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781652:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699023.781654:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699023.781660:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699023.781663:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699023.781669:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699023.781674:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699023.781679:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699023.781692:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699023.781696:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1787699023.781697:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126411776:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787699023.781699:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699023.781705:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6924 00000100:00080000:0.0:1787699023.781709:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699023.781712:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781715:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781717:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699023.781720:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699023.781722:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699023.781726:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781733:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699023.781737:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781738:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699023.781743:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699023.781749:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699023.781754:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000800:00000100:2.0:1787699023.781767:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787699023.781775:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699023.781781:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699023.781787:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699023.781795:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699023.781797:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699023.781798:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699023.781801:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6924 00000100:00080000:0.0:1787699023.781804:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699023.781807:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781809:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781811:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00100000:2.0:1787699023.781813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699023/real 1787699023] req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699023.781814:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699023.781819:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781829:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699023.781832:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699023.781836:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1787699023.781837:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699023.781841:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699023.781849:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699023.781851:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787699023.781869:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787699023.781870:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699023.781875:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699023.781882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699023.781909:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699023.781916:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1787699023.781917:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126411776:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787699023.781919:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699023.781925:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6924 00000100:00000040:2.0:1787699023.781926:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126411776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699023.781931:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699023.781938:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699023.781938:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699023.781944:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699023.781948:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699023.781950:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699023.781954:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699023.781955:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699023.781964:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:2.0:1787699023.781965:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411904:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787699023.781969:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699023.781973:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699023.781977:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:2.0:1787699023.781981:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699023.781982:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699023.781990:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787699023.781993:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699023.782003:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787699023.782003:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699023.782013:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699023.782014:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6934) 00000100:00000040:2.0:1787699023.782027:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:2.0:1787699023.782034:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699023/real 1787699023] req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782055:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699023.782061:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699023.782064:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699023.782067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782074:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126411904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699023.782079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699023.782104:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782111:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699023.782121:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699023.782128:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782135:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:2.0:1787699023.782141:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699023/real 1787699023] req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782150:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699023.782165:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699023.782167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699023.782171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782177:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699023.782182:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782190:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699023.782202:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782209:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699023.782219:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699023.782224:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782230:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:2.0:1787699023.782234:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699023/real 1787699023] req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782246:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699023.782251:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699023.782252:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699023.782254:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782258:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699023.782262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126411904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699023.782267:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699023.782271:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126412032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699023.782279:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:2.0:1787699023.782282:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126412160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699023.782286:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:2.0:1787699023.782290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782295:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699023.782301:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782306:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:2.0:1787699023.782311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699023.782315:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699048 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782320:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:2.0:1787699023.782323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699023/real 1787699023] req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782330:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699023.782335:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699023.782339:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787699023.782342:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699023.782345:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699023.782353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:2.0:1787699023.782358:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126412288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699048 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699023.782365:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699024.293534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699024.293966:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699024.293978:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699024.421579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699025.317476:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699025.317487:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699025.317494:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699025.445580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699025.701527:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301599087, at 4301599104 00000100:00000040:3.0:1787699025.701527:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301599087, at 4301599104 00000100:00100000:1.0:1787699026.341523:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699026.341523:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699026.341523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699026.469556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699027.365530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699027.365530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699027.365530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699027.493620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699028.389532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699028.389532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699028.389623:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699028.517584:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699028.901508:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699028.901515:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699028.901518:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699028.901526:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6929 00000100:00080000:2.0:1787699028.901534:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699028.901537:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901540:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901542:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699028.901546:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699028.901549:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901550:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901553:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699028.901555:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901566:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699028.901586:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699028.901602:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699028.901619:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699028.901622:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699028.901623:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699028.901626:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6929 00000100:00080000:2.0:1787699028.901629:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699028.901630:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901632:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901633:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699028.901635:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699028.901637:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901640:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699028.901641:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699028.901648:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699028.901650:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699028.901659:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699028.901669:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.901677:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699028.901680:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699028.901690:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699028.901693:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699028.901700:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6929 00000100:00100000:1.0:1787699028.901703:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126412416:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787699028.901706:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699028.901712:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901714:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901716:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699028.901718:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699028.901721:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901724:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901726:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699028.901729:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699028.901734:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:1.0:1787699028.901736:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699028.901742:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787699028.901752:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787699028.901757:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699028.901768:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699028.901778:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699028.901781:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699028.901783:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699028.901787:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6929 00000800:00000100:1.0:1787699028.901791:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787699028.901794:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699028.901800:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901804:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901806:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699028.901809:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699028.901821:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699028.901825:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901831:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901835:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699028.901840:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699028.901845:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699028.901848:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699028.901857:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699028.901864:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.901868:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699028/real 1787699028] req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699028.901873:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699028.901879:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699028.901882:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699028.901889:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6929 00000100:00080000:2.0:1787699028.901893:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699028.901896:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:1.0:1787699028.901900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699028.901900:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901904:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699028.901907:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699028.901913:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699028.901917:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699028.901921:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699028.901926:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699028.901933:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699028.901937:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787699028.901941:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699028.901945:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787699028.901962:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787699028.901968:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699028.901971:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787699028.901980:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699028.901986:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6940) 00000100:00100000:1.0:1787699028.901991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126412416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787699028.902000:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126412416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699028.902010:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787699028.902025:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902034:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787699028.902046:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902053:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787699028.902065:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699028.902072:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902081:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:1.0:1787699028.902087:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699028/real 1787699028] req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902110:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699028.902117:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699028.902120:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787699028.902124:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902131:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787699028.902136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902143:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787699028.902160:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902167:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787699028.902176:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699028.902182:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902190:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:1.0:1787699028.902195:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699028/real 1787699028] req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902210:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699028.902225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699028.902228:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787699028.902231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902237:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787699028.902242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902249:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787699028.902259:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902265:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787699028.902274:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699028.902280:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902288:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:1.0:1787699028.902292:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699028/real 1787699028] req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902307:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902314:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699028.902322:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699028.902324:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787699028.902328:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787699028.902339:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787699028.902355:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902361:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:1.0:1787699028.902370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699028.902376:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699053 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902384:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787699028.902388:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699028/real 1787699028] req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902398:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699028.902406:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699028.902412:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699028.902415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787699028.902446:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699028.902455:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126412928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:1.0:1787699028.902460:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126412544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699028.902467:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787699028.902473:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126412672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699028.902479:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:1.0:1787699028.902484:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126412800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699028.902490:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:1.0:1787699028.902495:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126412928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699053 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699028.902501:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699029.413549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699029.413549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699029.413669:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699029.541626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699030.437511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699030.437511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699030.437558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699030.565615:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699030.821511:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301604207, at 4301604224 00000100:00000040:3.0:1787699030.821527:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301604207, at 4301604224 00000100:00100000:3.0:1787699031.461535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699031.461536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699031.461556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699031.589636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699032.485521:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699032.485522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699032.485522:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699032.613587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699033.509498:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699033.509498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699033.509499:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699033.637596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699034.021556:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.021565:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699034.021569:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.021579:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6935 00000100:00080000:1.0:1787699034.021590:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699034.021593:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699034.021597:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.021600:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699034.021605:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699034.021609:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699034.021611:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.021614:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699034.021618:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699034.021632:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699034.021655:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699034.021670:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699034.021695:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.021699:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699034.021701:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.021706:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6935 00000100:00080000:1.0:1787699034.021710:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699034.021713:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699034.021717:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.021720:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699034.021735:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699034.021741:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699034.021743:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699034.021746:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.021750:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699034.021756:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:3.0:1787699034.021760:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126413056:192.168.201.136@tcp:250:kworker.0 00000008:00100000:1.0:1787699034.021767:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699034.021773:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699034.021786:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699034.021787:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699034.021799:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699034.021827:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:3.0:1787699034.021832:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699034.021834:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699034.021837:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.021842:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6935 00000100:00080000:1.0:1787699034.021847:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699034.021850:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699034.021853:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.021855:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699034.021858:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699034.021861:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699034.021863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.021867:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000800:00000100:3.0:1787699034.021868:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699034.021873:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699034.021879:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699034.021885:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699034.021889:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699034.021898:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699034.021911:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699034.021919:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699034.021922:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699034.021925:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787699034.021926:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699034.021934:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6935 00000100:00080000:1.0:1787699034.021940:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787699034.021942:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699034.021946:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699034.021954:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699034.021957:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699034.021958:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699034.021962:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00080000:3.0:1787699034.021967:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:1.0:1787699034.021969:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:3.0:1787699034.021973:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787699034.021976:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699034.021979:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699034.021981:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699034.021989:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:3.0:1787699034.021989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126413056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699034.021997:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126413056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699034.021999:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699034.022008:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:3.0:1787699034.022008:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699034.022019:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699034.022025:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699034.022033:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699034.022043:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787699034.022043:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413184:192.168.201.136@tcp:38:kworker.0 00000100:00080000:1.0:1787699034.022051:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699034.022052:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699034.022055:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6935 00000100:00080000:1.0:1787699034.022059:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699034.022064:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699034.022064:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699034.022067:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.022070:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699034.022073:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699034.022077:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699034.022078:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699034.022081:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699034.022085:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699034.022092:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:3.0:1787699034.022096:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:1.0:1787699034.022098:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699034.022106:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787699034.022111:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699034.022119:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022121:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699034.022138:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6945) 00000100:00000040:3.0:1787699034.022141:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699034.022149:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022162:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022171:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699034.022179:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699034.022182:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699034.022186:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022195:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699034.022201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699034.022221:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022229:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699034.022247:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699034.022255:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022263:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699034.022268:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022289:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699034.022297:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699034.022300:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699034.022304:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022312:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699034.022317:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126413184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699034.022324:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699034.022330:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126413312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699034.022337:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699034.022344:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699034.022374:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022382:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699034.022393:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699034.022400:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022409:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699034.022415:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022458:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699034.022473:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699034.022475:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699034.022477:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699034.022485:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022490:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699034.022503:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022507:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699034.022515:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699034.022519:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699059 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022524:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699034.022528:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699034/real 1787699034] req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022534:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699034.022539:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699034.022543:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699034.022545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699034.022547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699034.022551:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699034.022554:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126413440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699034.022558:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699034.022562:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126413568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699059 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699034.022579:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699034.533527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699034.533527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699034.533528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699034.661580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699035.557529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699035.557529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699035.557529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699035.685623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699035.941532:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301609327, at 4301609344 00000100:00000040:3.0:1787699035.941532:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301609327, at 4301609344 00000100:00100000:1.0:1787699036.581524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699036.581524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699036.581526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699036.709549:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699037.605535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699037.605535:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699037.605535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699037.733735:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699038.629502:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699038.629502:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699038.629506:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699038.757577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699039.141528:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.141537:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699039.141540:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.141548:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6940 00000100:00080000:0.0:1787699039.141557:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699039.141559:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141562:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141565:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699039.141569:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699039.141572:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141575:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141577:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699039.141580:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141591:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699039.141611:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699039.141622:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699039.141646:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.141650:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699039.141651:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.141654:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6940 00000100:00080000:0.0:1787699039.141658:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699039.141660:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141662:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141663:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699039.141665:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699039.141667:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141669:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141671:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699039.141672:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699039.141680:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699039.141683:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699039.141690:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699039.141699:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699039.141701:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699039.141731:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.141733:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699039.141735:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.141739:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6940 00000100:00100000:3.0:1787699039.141739:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126413696:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787699039.141741:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699039.141745:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141750:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141752:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699039.141753:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699039.141755:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141756:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141758:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699039.141759:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699039.141762:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699039.141765:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699039.141777:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699039.141778:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699039.141811:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141814:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699039.141824:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.141828:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699039.141830:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699039.141833:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6940 00000100:00080000:0.0:1787699039.141836:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699039.141838:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141843:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:3.0:1787699039.141844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141845:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699039.141849:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699039.141851:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141854:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141855:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699039.141857:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:3.0:1787699039.141861:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699039.141862:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699039.141866:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699039.141874:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699039.141884:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699039.141889:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699039.141891:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699039.141893:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787699039.141896:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699039.141898:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6940 00000100:00080000:0.0:1787699039.141904:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699039.141909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141913:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141916:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699039.141918:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699039.141920:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699039.141922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699039.141924:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:3.0:1787699039.141925:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699039.141930:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:3.0:1787699039.141937:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699039.141938:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699039.141948:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:3.0:1787699039.141949:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699039.141954:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:0.0:1787699039.141961:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699039.141978:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.141978:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699039.141999:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6950) 00000100:00100000:3.0:1787699039.142002:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126413696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699039.142013:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126413696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699039.142021:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699039.142035:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142042:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699039.142052:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142057:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699039.142066:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699039.142070:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142076:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699039.142080:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699039.142105:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699039.142114:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699039.142118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699039.142132:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142138:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699039.142147:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142152:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699039.142160:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699039.142164:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142178:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699039.142182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142189:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142194:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699039.142199:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699039.142201:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699039.142203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142212:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126413952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699039.142215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699039.142228:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142233:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699039.142239:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699039.142243:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142248:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699039.142251:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142258:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142262:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699039.142266:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699039.142268:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699039.142270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142276:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699039.142286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142294:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699039.142307:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142314:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699039.142326:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699039.142333:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699064 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142341:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699039.142347:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699039/real 1787699039] req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142357:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699039.142365:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699039.142372:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699039.142375:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699039.142379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699039.142386:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699039.142391:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126413824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699039.142412:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699039.142450:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126413952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699039.142459:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699039.142466:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126414080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699039.142473:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699039.142477:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126414208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699064 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699039.142484:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699039.653527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699039.653527:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699039.653528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699039.781572:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699040.677517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699040.677517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699040.677518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699040.805577:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699041.061503:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301614447, at 4301614464 00000100:00000040:2.0:1787699041.061505:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301614447, at 4301614464 00000100:00100000:1.0:1787699041.701516:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699041.701516:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699041.701516:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699041.829562:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699042.725482:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699042.725483:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699042.725490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699042.853613:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699043.749517:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699043.749517:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699043.749517:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699043.879760:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699044.261500:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699044.261509:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699044.261514:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699044.261523:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6945 00000100:00080000:1.0:1787699044.261532:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699044.261534:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261537:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261540:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699044.261544:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699044.261546:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261548:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261559:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699044.261563:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261573:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699044.261592:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699044.261602:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699044.261619:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699044.261622:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699044.261623:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699044.261626:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6945 00000100:00080000:1.0:1787699044.261629:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699044.261630:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261632:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261634:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699044.261635:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699044.261637:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261639:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261640:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699044.261641:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699044.261648:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699044.261651:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699044.261656:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699044.261665:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699044.261671:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699044.261672:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699044.261673:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699044.261675:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6945 00000100:00080000:1.0:1787699044.261677:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699044.261679:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261680:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261681:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699044.261682:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699044.261684:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261687:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261695:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:3.0:1787699044.261697:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699044.261702:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699044.261714:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699044.261726:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699044.261733:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126414336:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787699044.261739:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699044.261749:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699044.261763:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699044.261771:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699044.261772:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699044.261781:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6945 00000100:00000040:3.0:1787699044.261782:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699044.261788:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699044.261800:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699044.261807:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261808:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261812:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699044.261814:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699044.261816:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261818:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261819:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699044.261821:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699044.261826:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699044.261830:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699044.261839:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699044.261848:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787699044.261863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699044.261873:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699044.261878:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699044.261881:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699044.261886:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6945 00000100:00080000:1.0:1787699044.261890:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699044.261891:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699044.261896:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261904:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261910:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699044.261915:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699044.261922:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699044.261924:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699044.261925:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699044.261927:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699044.261931:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699044.261934:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699044.261940:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699044.261964:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699044.261984:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6955) 00000100:00100000:3.0:1787699044.262017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699044/real 1787699044] req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262036:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262043:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699044.262050:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699044.262053:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699044.262057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262065:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126414336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699044.262073:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699044.262110:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262118:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699044.262132:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699044.262140:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262151:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699044.262157:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699044/real 1787699044] req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262192:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699044.262208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699044.262211:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699044.262215:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262221:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699044.262227:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126414336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699044.262237:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699044.262247:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126414464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699044.262253:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699044.262266:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699044.262287:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262294:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699044.262306:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699044.262320:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262330:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699044.262336:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699044/real 1787699044] req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262347:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699044.262364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699044.262368:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699044.262372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699044.262386:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262394:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699044.262406:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262413:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699044.262464:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699044.262473:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262493:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699044.262500:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699044/real 1787699044] req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262510:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262519:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699044.262526:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699044.262530:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699044.262534:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699044.262546:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262568:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699044.262588:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262595:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699044.262607:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699044.262613:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699069 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262621:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699044.262633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699044/real 1787699044] req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262644:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699044.262652:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699044.262659:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699044.262662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699044.262666:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699044.262673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126414848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699044.262679:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126414592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699044.262687:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699044.262693:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126414720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699044.262699:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699044.262704:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126414848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699069 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699044.262710:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699044.773536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699044.773552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699044.773592:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699044.901571:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699045.797545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699045.797545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699045.797556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699045.925556:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699046.181504:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301619567, at 4301619584 00000100:00000040:3.0:1787699046.181504:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301619567, at 4301619584 00000100:00100000:1.0:1787699046.821542:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699046.821543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699046.821543:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699046.949599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699047.845528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699047.845528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699047.845559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699047.973587:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699048.869551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699048.869551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699048.869571:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699048.997624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699049.381586:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699049.381594:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699049.381597:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699049.381605:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6950 00000100:00080000:0.0:1787699049.381622:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699049.381625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.381627:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381630:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699049.381634:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699049.381636:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.381638:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381640:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699049.381642:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381655:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699049.381674:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699049.381685:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699049.381705:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699049.381707:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699049.381708:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699049.381711:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6950 00000100:00080000:0.0:1787699049.381713:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699049.381718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.381722:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381724:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699049.381726:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699049.381729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.381732:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381735:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699049.381737:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699049.381746:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699049.381749:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699049.381758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699049.381764:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699049.381818:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699049.381827:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787699049.381828:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126414976:192.168.201.136@tcp:250:kworker.0 00000100:00080000:0.0:1787699049.381842:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699049.381845:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699049.381852:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6950 00000100:00080000:0.0:1787699049.381856:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699049.381860:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.381863:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381865:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699049.381867:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699049.381871:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699049.381873:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.381891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699049.381894:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381900:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699049.381905:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699049.381909:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699049.381913:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699049.381921:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699049.381929:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699049.381936:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699049.381938:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699049.381940:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699049.381948:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6950 00000800:00000100:3.0:1787699049.381948:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699049.381958:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699049.381961:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.381963:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381965:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699049.381967:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699049.381969:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699049.381972:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699049.381977:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.381987:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699049.381998:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699049.382004:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699049.382009:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699049.382019:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699049.382026:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699049/real 1787699049] req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699049.382032:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699049.382081:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699049.382083:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699049.382084:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699049.382087:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6950 00000100:00080000:0.0:1787699049.382089:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699049.382096:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:3.0:1787699049.382096:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699049.382105:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.382109:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699049.382113:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699049.382119:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699049.382126:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699049.382133:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699049.382139:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699049.382151:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699049.382156:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699049.382158:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699049.382165:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699049.382184:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787699049.382191:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699049.382197:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699049.382216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699049.382218:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6960) 00000100:00100000:3.0:1787699049.382230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126414976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699049.382236:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126414976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699049.382245:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699049.382260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699049.382293:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382298:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699049.382307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699049.382311:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382319:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699049.382322:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699049/real 1787699049] req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699049.382338:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699049.382340:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699049.382342:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382346:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699049.382350:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382355:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699049.382367:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382372:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699049.382378:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699049.382382:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382388:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699049.382391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699049/real 1787699049] req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382397:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699049.382407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699049.382409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699049.382411:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415232:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699049.382438:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382444:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699049.382452:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382460:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699049.382482:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699049.382488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382494:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699049.382497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699049/real 1787699049] req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382504:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699049.382514:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699049.382516:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699049.382518:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382522:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699049.382526:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382531:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699049.382538:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382542:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699049.382553:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699049.382557:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699074 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382563:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699049.382566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699049/real 1787699049] req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382573:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699049.382578:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699049.382582:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699049.382584:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699049.382585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699049.382589:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699049.382592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126415104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699049.382596:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699049.382600:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126415232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699049.382604:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699049.382607:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126415360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699049.382611:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699049.382617:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126415488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699074 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699049.382622:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699049.893514:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699049.893514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699049.893514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699050.021653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699050.917498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699050.917498:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699050.917566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699051.045640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699051.301551:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301624687, at 4301624704 00000100:00000040:3.0:1787699051.301551:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301624687, at 4301624704 00000100:00100000:1.0:1787699051.941548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699051.941548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699051.941552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699052.069608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699052.965511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699052.965511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699052.965512:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699053.093642:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699053.989530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699053.989530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699053.989628:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699054.117605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699054.501563:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699054.501577:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699054.501582:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699054.501597:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6955 00000100:00080000:0.0:1787699054.501610:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699054.501615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501620:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501624:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699054.501629:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699054.501634:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501636:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501638:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699054.501641:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501657:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699054.501682:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699054.501695:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699054.501725:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699054.501729:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699054.501731:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699054.501734:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6955 00000100:00080000:0.0:1787699054.501738:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699054.501741:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501743:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501745:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699054.501748:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699054.501751:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501755:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501757:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699054.501761:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699054.501771:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:3.0:1787699054.501775:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699054.501778:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699054.501794:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126415616:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787699054.501795:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699054.501814:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699054.501823:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:3.0:1787699054.501834:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699054.501835:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699054.501838:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699054.501846:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6955 00000100:00080000:0.0:1787699054.501852:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699054.501852:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501859:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501862:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501865:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699054.501867:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699054.501871:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501874:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501876:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699054.501878:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699054.501884:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699054.501890:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699054.501901:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699054.501912:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699054.501920:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699054.501923:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699054.501925:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699054.501930:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6955 00000100:00080000:0.0:1787699054.501934:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699054.501938:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501941:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501944:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699054.501947:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699054.501951:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.501954:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.501959:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:3.0:1787699054.501959:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699054.501964:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699054.501974:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699054.501977:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415744:192.168.201.136@tcp:38:kworker.0 00000020:00000040:0.0:1787699054.501982:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699054.501996:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699054.501998:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699054.502017:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699054.502024:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699054.502035:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699054.502038:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00000040:3.0:1787699054.502038:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:0.0:1787699054.502039:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699054.502048:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6955 00000100:00000040:3.0:1787699054.502052:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699054.502053:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699054.502058:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.502062:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.502067:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699054.502069:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699054.502072:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699054.502074:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699054.502076:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699054.502080:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:3.0:1787699054.502086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415872:192.168.201.136@tcp:38:kworker.0 00000008:00100000:0.0:1787699054.502087:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699054.502095:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699054.502106:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699054.502122:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699054.502130:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699054.502134:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6965) 00000100:00000040:3.0:1787699054.502143:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699054.502158:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787699054.502167:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699054.502174:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699054.502186:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699054.502192:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699054.502199:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787699054.502202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699054.502208:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699054.502214:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699054.502218:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699054.502222:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787699055.013504:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699055.013504:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699055.013534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699055.141618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699056.037584:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699056.037587:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699056.037589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699056.165593:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699056.421542:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301629807, at 4301629824 00000100:00000040:2.0:1787699056.421555:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301629807, at 4301629824 00000100:00100000:1.0:1787699057.061563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699057.061563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699057.061564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699057.189690:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787699057.573750:0:2388:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:1.0:1787699057.573815:0:2388:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787699057.573835:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699057.573874:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699057.573892:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699057.573900:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699057.573923:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699057.573932:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699057.573942:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699057.573952:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699057.573963:0:2388:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699057.573974:0:2388:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699079 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.573983:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699054/real 1787699057] req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699057.574062:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699057.574074:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699057.574079:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699057.574085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574093:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416000:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787699057.574101:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699054/real 1787699057] req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699057.574127:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699057.574134:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699057.574137:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699057.574141:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574148:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416128:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787699057.574154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699054/real 1787699057] req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574180:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699057.574189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699057.574197:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699057.574200:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699057.574204:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574211:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415744:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787699057.574217:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699054/real 1787699057] req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574227:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699057.574235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699057.574243:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699057.574246:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699057.574250:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126415872:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787699057.574271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699054/real 1787699057] req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574284:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699057.574294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699057.574301:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699057.574304:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699057.574308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699057.574316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126415616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699057.574322:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126416000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699057.574377:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699057.574488:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126416128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699057.574498:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699057.574505:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126415744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699057.574511:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699057.574516:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126415872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699057.574524:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699057.574537:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126415616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699079 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699057.574544:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699058.085530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699058.085530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699058.085538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699058.213673:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699059.109557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699059.109561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699059.109562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699059.237630:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699059.621564:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699059.621575:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699059.621578:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699059.621590:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6960 00000100:00080000:2.0:1787699059.621601:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699059.621603:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621606:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621609:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699059.621614:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699059.621617:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621621:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699059.621623:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621633:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699059.621664:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699059.621681:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699059.621706:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699059.621709:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699059.621712:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699059.621731:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6960 00000100:00080000:2.0:1787699059.621736:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699059.621737:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699059.621742:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621750:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621755:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699059.621758:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00100000:0.0:1787699059.621762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126416256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787699059.621765:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621770:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621773:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699059.621775:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699059.621786:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699059.621792:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699059.621799:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699059.621805:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699059.621816:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621826:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699059.621836:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699059.621840:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699059.621842:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699059.621845:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6960 00000100:00080000:2.0:1787699059.621853:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699059.621855:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621858:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621860:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000800:00000100:0.0:1787699059.621865:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621865:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699059.621872:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621875:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621878:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699059.621882:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699059.621885:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699059.621889:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699059.621898:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699059.621910:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699059.621920:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699059.621928:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787699059.621931:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699059/real 1787699059] req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699059.621933:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699059.621937:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699059.621946:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6960 00000100:00080000:2.0:1787699059.621950:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699059.621959:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699059.621965:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.621973:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699059.621974:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699059.621977:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699059.621981:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699059.621988:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:0.0:1787699059.621993:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787699059.621993:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.621999:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699059.622002:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00080000:2.0:1787699059.622010:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699059.622011:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699059.622020:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699059.622025:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126416256:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787699059.622029:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699059.622036:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126416256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699059.622043:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000020:00000040:0.0:1787699059.622051:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:2.0:1787699059.622057:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699059.622067:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699059.622071:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699059.622073:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787699059.622075:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699059.622080:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6960 00000100:00080000:2.0:1787699059.622087:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699059.622092:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787699059.622096:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:2.0:1787699059.622098:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.622104:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699059.622107:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699059.622110:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699059.622113:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699059.622115:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:0.0:1787699059.622118:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699059.622121:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699059.622130:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787699059.622133:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787699059.622139:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:0.0:1787699059.622149:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699059.622157:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699059.622163:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699059.622170:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:2.0:1787699059.622176:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622181:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699059/real 1787699059] req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699059.622200:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6970) 00000100:00100000:0.0:1787699059.622203:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622218:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699059.622226:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699059.622230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699059.622234:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622242:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416384:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699059.622247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699059.622269:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622277:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699059.622301:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699059.622310:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622319:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699059.622324:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699059/real 1787699059] req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622334:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699059.622350:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699059.622352:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699059.622356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622368:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699059.622373:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622381:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699059.622394:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622401:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699059.622411:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699059.622443:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622456:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699059.622463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699059/real 1787699059] req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622475:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622483:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699059.622491:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699059.622496:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699059.622500:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622508:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699059.622515:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126416384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699059.622524:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699059.622540:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126416512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699059.622547:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699059.622551:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126416640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699059.622559:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699059.622567:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622575:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699059.622586:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622594:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699059.622605:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699059.622612:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699084 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622620:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699059.622624:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699059/real 1787699059] req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622634:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699059.622642:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699059.622648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699059.622652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699059.622655:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699059.622661:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126416768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699059.622667:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126416768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699084 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699059.622673:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699060.133503:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699060.133503:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699060.133503:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699060.261618:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699061.157534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699061.157534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699061.157534:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699061.285607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699061.541542:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301634927, at 4301634944 00000100:00000040:3.0:1787699061.541542:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301634927, at 4301634944 00000100:00100000:2.0:1787699062.181532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699062.181533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699062.181805:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699062.309619:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699063.205567:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699063.205567:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699063.205567:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699063.333626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699064.229528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699064.229528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699064.229536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699064.357604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699064.741571:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699064.741584:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699064.741589:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699064.741603:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6965 00000100:00080000:2.0:1787699064.741616:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699064.741619:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.741624:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741627:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699064.741633:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699064.741636:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.741639:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741642:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699064.741645:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741663:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699064.741695:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699064.741711:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699064.741736:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699064.741739:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699064.741740:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699064.741745:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6965 00000100:00080000:2.0:1787699064.741749:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699064.741751:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.741754:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741757:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699064.741759:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699064.741762:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.741766:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741768:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699064.741771:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699064.741781:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699064.741786:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699064.741795:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699064.741801:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699064.741822:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.741826:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126416896:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787699064.741840:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699064.741844:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699064.741846:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699064.741850:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6965 00000100:00080000:2.0:1787699064.741854:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699064.741857:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.741859:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741861:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699064.741864:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699064.741870:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699064.741871:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699064.741875:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741881:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699064.741886:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699064.741891:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699064.741898:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699064.741901:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:2.0:1787699064.741911:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699064.741922:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699064.741930:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699064.741934:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699064.741936:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699064.741941:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6965 00000100:00080000:2.0:1787699064.741944:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699064.741947:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:0.0:1787699064.741952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699064.741953:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741958:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699064.741961:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699064.741964:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.741967:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.741970:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699064.741973:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699064.741977:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699064.741980:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699064.741988:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699064.742003:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699064.742013:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699064.742021:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699064.742025:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699064.742027:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699064.742030:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6965 00000100:00080000:2.0:1787699064.742035:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699064.742036:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699064/real 1787699064] req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699064.742041:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.742048:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.742052:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699064.742056:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699064.742060:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699064.742063:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699064.742065:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00100000:0.0:1787699064.742069:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699064.742070:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699064.742078:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699064.742085:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699064.742087:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699064.742099:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787699064.742104:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699064.742105:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699064.742114:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699064.742129:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126416896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787699064.742148:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6975) 00000100:00000040:0.0:1787699064.742149:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126416896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699064.742163:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699064.742181:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742191:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699064.742219:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742227:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699064.742241:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699064.742248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742265:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699064.742271:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699064/real 1787699064] req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699064.742300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699064.742303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699064.742307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699064.742321:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742329:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699064.742342:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742350:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699064.742362:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699064.742369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742378:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699064.742385:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699064/real 1787699064] req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742396:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742405:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699064.742413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699064.742416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699064.742449:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699064.742463:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742471:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699064.742484:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742492:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699064.742502:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699064.742509:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742519:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699064.742525:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699064/real 1787699064] req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742543:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742552:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699064.742559:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699064.742562:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699064.742566:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742574:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699064.742579:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742587:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699064.742599:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742607:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699064.742618:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699064.742625:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699089 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742646:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699064.742653:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699064/real 1787699064] req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742663:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699064.742673:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699064.742680:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699064.742683:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699064.742687:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699064.742694:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699064.742700:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126417024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699064.742708:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699064.742714:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126417152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699064.742721:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699064.742726:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126417280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699064.742732:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699064.742737:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126417408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699089 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699064.742744:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699065.253529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699065.253529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699065.253544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699065.381591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699066.277521:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699066.277521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699066.277524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699066.405712:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699066.661538:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301640047, at 4301640064 00000100:00000040:3.0:1787699066.661538:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301640047, at 4301640064 00000100:00100000:0.0:1787699067.301498:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699067.301498:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699067.301507:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699067.429646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699068.325558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699068.325558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699068.325566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699068.453707:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699069.349536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699069.349536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699069.349536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699069.477599:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699069.861541:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699069.861562:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699069.861567:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699069.861579:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6970 00000100:00080000:2.0:1787699069.861593:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699069.861597:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.861602:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861606:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699069.861612:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699069.861615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.861618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861621:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699069.861625:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861643:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699069.861677:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699069.861694:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699069.861721:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699069.861725:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699069.861729:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699069.861733:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6970 00000100:00080000:2.0:1787699069.861737:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699069.861741:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.861744:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861747:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699069.861750:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699069.861753:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.861757:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861762:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:0.0:1787699069.861763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699069.861768:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787699069.861782:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126417536:192.168.201.136@tcp:250:kworker.0 00000008:00100000:2.0:1787699069.861783:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699069.861791:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699069.861802:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699069.861814:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699069.861817:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.861828:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787699069.861837:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699069.861840:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699069.861843:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699069.861848:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6970 00000100:00080000:2.0:1787699069.861852:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699069.861856:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.861859:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861861:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699069.861864:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000800:00000100:0.0:1787699069.861869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699069.861871:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.861876:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861878:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699069.861882:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699069.861889:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699069.861889:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699069.861898:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699069.861910:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699069.861921:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.861923:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699069/real 1787699069] req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699069.861943:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699069.861948:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787699069.861951:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699069.861952:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699069.861961:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6970 00000100:00080000:2.0:1787699069.861969:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699069.861971:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699069.861975:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00080000:0.0:1787699069.861983:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787699069.861983:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.861987:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00080000:0.0:1787699069.861990:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:2.0:1787699069.861992:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699069.861996:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699069.862000:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.862007:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787699069.862009:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126417536:192.168.201.136@tcp:250:kworker.0 00000020:00000040:2.0:1787699069.862013:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:0.0:1787699069.862018:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126417536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699069.862021:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699069.862027:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699069.862037:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000020:00000040:0.0:1787699069.862042:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699069.862060:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699069.862065:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699069.862082:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862086:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417664:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787699069.862100:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699069.862104:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699069.862106:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787699069.862111:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699069.862113:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6970 00000100:00080000:2.0:1787699069.862133:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699069.862136:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699069.862139:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.862143:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699069.862147:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699069.862152:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699069.862156:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699069.862159:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699069.862162:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699069.862165:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699069.862170:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1787699069.862173:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787699069.862178:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699069.862181:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699069.862191:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699069.862192:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699069.862201:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699069/real 1787699069] req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699069.862210:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862214:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862232:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699069.862234:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6980) 00000100:00080000:0.0:1787699069.862245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699069.862247:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699069.862252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862257:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699069.862263:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862272:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699069.862281:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862286:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699069.862293:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699069.862299:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862308:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699069.862314:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699069/real 1787699069] req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862333:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699069.862341:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699069.862345:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699069.862348:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862353:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699069.862357:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126417664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699069.862362:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699069.862367:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126417792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699069.862371:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699069.862376:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862382:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699069.862392:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862396:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699069.862403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699069.862407:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862414:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699069.862438:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699069/real 1787699069] req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862446:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699069.862456:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699069.862458:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699069.862460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862464:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126417920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699069.862467:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699069.862479:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862484:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699069.862490:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699069.862494:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699094 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862499:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699069.862503:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699069/real 1787699069] req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862509:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699069.862513:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699069.862517:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699069.862518:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699069.862521:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699069.862525:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699069.862528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126417920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699069.862536:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699069.862539:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126418048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699094 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699069.862543:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699070.373508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699070.373509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699070.373511:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699070.501580:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699071.397513:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699071.397513:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699071.397515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699071.525538:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699071.781511:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301645167, at 4301645184 00000100:00000040:3.0:1787699071.781511:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301645167, at 4301645184 00000100:00100000:0.0:1787699072.421489:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699072.421489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699072.421490:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699072.549624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699073.445529:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699073.445529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699073.445529:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699073.573563:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699074.469556:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699074.469557:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699074.469557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699074.597600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699074.981578:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699074.981587:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699074.981591:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699074.981601:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6975 00000100:00080000:0.0:1787699074.981612:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699074.981615:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981618:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981621:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699074.981625:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699074.981628:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981629:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981632:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699074.981634:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981649:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699074.981674:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699074.981685:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699074.981706:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699074.981709:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699074.981710:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699074.981713:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6975 00000100:00080000:0.0:1787699074.981715:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699074.981716:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981718:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981719:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699074.981721:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699074.981723:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981726:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981727:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699074.981729:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699074.981738:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699074.981741:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699074.981746:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699074.981754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699074.981755:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699074.981773:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699074.981777:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699074.981778:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699074.981780:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6975 00000100:00080000:0.0:1787699074.981782:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787699074.981785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126418176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699074.981787:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981792:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981793:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699074.981795:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699074.981796:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981798:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981799:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699074.981800:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699074.981803:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699074.981806:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699074.981814:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699074.981819:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699074.981824:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699074.981832:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699074.981834:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699074.981834:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699074.981837:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6975 00000100:00080000:0.0:1787699074.981839:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699074.981841:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981843:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981846:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699074.981849:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699074.981854:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981858:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981862:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:3.0:1787699074.981863:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699074.981866:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699074.981871:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699074.981873:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699074.981878:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699074.981884:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699074.981888:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699074.981890:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699074.981891:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699074.981892:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6975 00000100:00080000:0.0:1787699074.981895:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699074.981896:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981898:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981899:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699074.981900:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699074.981902:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699074.981903:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699074.981904:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699074.981906:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699074.981908:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699074.981912:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000800:00000100:3.0:1787699074.981913:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:0.0:1787699074.981918:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699074.981925:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699074.981933:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6986) 00000100:00000040:3.0:1787699074.981937:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.981980:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699074/real 1787699074] req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.981997:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982013:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699074.982021:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699074.982025:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699074.982038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982046:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126418176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699074.982053:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126418176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699074.982063:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699074.982079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982087:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699074.982101:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982118:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699074.982131:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699074.982138:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982148:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699074.982153:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699074/real 1787699074] req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982178:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699074.982185:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699074.982188:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699074.982192:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982199:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699074.982205:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699074.982224:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982231:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699074.982249:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699074.982263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982273:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699074.982279:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699074/real 1787699074] req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982303:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982311:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699074.982319:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699074.982322:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699074.982326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982334:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699074.982340:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982348:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699074.982358:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982362:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699074.982370:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699074.982384:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982403:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699074.982409:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699074/real 1787699074] req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982442:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982451:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699074.982466:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699074.982470:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699074.982474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982482:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699074.982488:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982500:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699074.982514:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982521:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699074.982533:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699074.982541:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699099 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982550:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699074.982569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699074/real 1787699074] req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982581:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699074.982590:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699074.982598:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699074.982601:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699074.982604:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699074.982612:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699074.982618:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126418304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699074.982627:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699074.982633:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126418432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699074.982640:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699074.982646:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126418560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699074.982653:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699074.982658:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126418688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699099 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699074.982664:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699075.493519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699075.493519:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699075.493520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699075.621608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699076.517547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699076.517553:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699076.517591:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699076.645606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699076.901514:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301650287, at 4301650304 00000100:00000040:3.0:1787699076.902184:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301650287, at 4301650304 00000100:00100000:1.0:1787699077.541575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699077.541578:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699077.541582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699077.669653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699078.565544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699078.565547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699078.565547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699078.693634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699079.589519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699079.589520:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699079.589625:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699079.717669:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699080.101566:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699080.101573:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699080.101577:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699080.101588:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6981 00000100:00080000:2.0:1787699080.101597:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699080.101601:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101605:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101608:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699080.101612:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699080.101614:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101616:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101618:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699080.101620:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101631:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699080.101659:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699080.101671:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699080.101692:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699080.101696:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699080.101697:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699080.101701:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6981 00000100:00080000:2.0:1787699080.101705:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699080.101707:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101713:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699080.101716:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699080.101720:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101723:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101726:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699080.101729:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699080.101739:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699080.101744:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699080.101755:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699080.101762:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699080.101803:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.101805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126418816:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787699080.101827:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699080.101830:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699080.101832:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699080.101837:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6981 00000100:00080000:2.0:1787699080.101842:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699080.101845:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101848:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101851:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699080.101853:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699080.101856:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699080.101863:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101875:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699080.101881:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101882:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699080.101888:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699080.101893:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699080.101898:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699080.101911:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699080.101921:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699080.101930:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699080.101933:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699080.101939:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787699080.101955:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787699080.101956:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6981 00000100:00080000:2.0:1787699080.101966:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699080.101969:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101971:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101973:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699080.101975:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699080.101978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.101980:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.101983:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699080.101985:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699080.101991:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:3.0:1787699080.101992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699080.101996:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699080.102009:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699080.102017:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699080.102024:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699080.102026:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699080.102028:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699080.102031:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6981 00000100:00080000:2.0:1787699080.102036:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699080.102042:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:3.0:1787699080.102046:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699080] req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699080.102051:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.102063:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699080.102069:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699080.102073:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699080.102077:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699080.102078:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699080.102083:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:3.0:1787699080.102089:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699080.102090:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699080.102100:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699080.102114:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699080.102120:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787699080.102135:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787699080.102141:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699080.102147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699080.102164:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699080.102166:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6991) 00000100:00100000:3.0:1787699080.102207:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126418816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699080.102216:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126418816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699080.102227:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699080.102247:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102256:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699080.102272:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102280:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699080.102294:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699080.102302:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102312:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699080.102317:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699080] req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102329:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699080.102345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699080.102348:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699080.102359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102366:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126418944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699080.102372:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102379:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699080.102393:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102400:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699080.102412:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699080.102447:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102458:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699080.102463:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699080] req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102474:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699080.102490:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699080.102493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699080.102497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102527:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699080.102533:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102543:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699080.102558:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102566:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699080.102579:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699080.102585:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102594:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699080.102610:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699080] req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102622:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102631:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699080.102639:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699080.102643:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699080.102646:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699080.102665:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699080.102687:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102694:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699080.102705:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699080.102711:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699105 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102719:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699080.102724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699080/real 1787699080] req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102734:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699080.102742:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699080.102749:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699080.102752:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699080.102756:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699080.102763:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699080.102768:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126418944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699080.102786:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699080.102793:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126419072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699080.102799:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699080.102804:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126419200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699080.102811:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699080.102822:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126419328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699105 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699080.102828:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699080.613554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699080.613554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699080.613614:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699080.741666:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699081.637534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699081.637578:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699081.637591:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699081.765639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699082.021572:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301655407, at 4301655424 00000100:00000040:3.0:1787699082.021572:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301655407, at 4301655424 00000100:00100000:1.0:1787699082.661542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699082.661542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699082.661548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699082.789578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699083.685531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699083.685531:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699083.685575:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699083.813612:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699084.709515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699084.709515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699084.709515:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699084.837617:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699085.221560:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699085.221570:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699085.221574:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699085.221583:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6986 00000100:00080000:1.0:1787699085.221594:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699085.221598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221605:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699085.221610:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699085.221612:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221615:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221618:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699085.221622:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221637:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699085.221660:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699085.221671:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699085.221693:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699085.221695:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699085.221696:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699085.221699:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6986 00000100:00080000:1.0:1787699085.221702:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699085.221704:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221706:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221707:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699085.221709:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699085.221711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221713:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221714:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699085.221717:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699085.221725:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699085.221728:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699085.221735:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699085.221743:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699085.221749:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699085.221751:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699085.221752:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699085.221755:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6986 00000100:00080000:1.0:1787699085.221757:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699085.221759:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221760:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221762:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699085.221763:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699085.221765:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221773:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221777:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:3.0:1787699085.221777:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699085.221788:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699085.221794:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699085.221807:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699085.221824:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699085.221824:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126419456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787699085.221846:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699085.221852:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699085.221864:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699085.221866:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699085.221870:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6986 00000100:00080000:1.0:1787699085.221873:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699085.221875:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221877:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221878:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699085.221880:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699085.221881:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221888:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:3.0:1787699085.221895:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699085.221896:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699085.221903:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699085.221911:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699085.221922:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787699085.221923:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699085.221932:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699085.221938:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699085.221940:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699085.221940:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699085.221943:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6986 00000100:00080000:1.0:1787699085.221946:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699085.221948:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221950:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221951:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699085.221952:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699085.221953:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699085.221955:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699085.221956:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699085.221958:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699085.221960:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699085.221963:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699085.221966:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699085.221971:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699085.221978:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (6996) 00000800:00000100:3.0:1787699085.221989:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699085.222015:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222070:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699085/real 1787699085] req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222088:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699085.222108:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699085.222112:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699085.222116:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222124:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126419456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699085.222131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699085.222156:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222164:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699085.222175:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699085.222183:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222193:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699085.222198:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699085/real 1787699085] req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222217:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222225:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699085.222232:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699085.222236:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699085.222240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222259:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699085.222265:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126419456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699085.222278:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699085.222292:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126419584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699085.222299:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699085.222313:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222321:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699085.222334:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222341:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699085.222353:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699085.222366:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222376:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699085.222382:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699085/real 1787699085] req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222393:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222402:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699085.222410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699085.222413:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699085.222451:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222459:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699085.222465:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699085.222486:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222493:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699085.222504:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699085.222511:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222540:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699085.222546:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699085/real 1787699085] req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222564:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222573:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699085.222580:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699085.222583:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699085.222587:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222595:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699085.222601:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222609:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699085.222620:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222627:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699085.222638:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699085.222644:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699110 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222653:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699085.222663:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699085/real 1787699085] req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222673:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699085.222681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699085.222688:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699085.222691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699085.222695:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699085.222703:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126419968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699085.222708:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126419712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699085.222716:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699085.222722:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126419840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699085.222729:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699085.222734:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126419968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699110 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699085.222740:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699085.733577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699085.733577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699085.733582:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699085.861620:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699086.757570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699086.757570:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699086.757570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699086.885635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699087.141528:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301660527, at 4301660544 00000100:00000040:3.0:1787699087.141528:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301660527, at 4301660544 00000100:00100000:1.0:1787699087.781525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699087.781525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699087.781525:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699087.912976:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699088.805558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699088.805558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699088.805558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699088.933634:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699089.829505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699089.829505:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699089.829505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699089.957573:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699090.341515:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699090.341530:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699090.341535:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699090.341550:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6991 00000100:00080000:1.0:1787699090.341564:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699090.341568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341573:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341577:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699090.341583:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699090.341587:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341590:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341593:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699090.341597:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341613:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699090.341660:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699090.341676:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699090.341700:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699090.341703:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699090.341706:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699090.341710:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6991 00000100:00080000:1.0:1787699090.341714:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699090.341717:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341721:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341723:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699090.341727:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699090.341730:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341733:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341736:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699090.341738:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:3.0:1787699090.341743:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699090.341749:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699090.341755:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699090.341759:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126420096:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787699090.341793:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699090.341808:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699090.341809:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699090.341836:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:3.0:1787699090.341839:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699090.341840:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699090.341844:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699090.341847:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6991 00000100:00080000:1.0:1787699090.341850:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699090.341852:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341854:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341855:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699090.341857:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699090.341860:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341862:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341863:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787699090.341865:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699090.341869:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699090.341873:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699090.341883:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787699090.341889:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341898:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699090.341908:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:3.0:1787699090.341913:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699090.341914:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699090.341918:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699090.341925:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6991 00000100:00080000:1.0:1787699090.341931:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699090.341935:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341938:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341940:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699090.341943:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699090.341946:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.341949:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.341952:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699090.341955:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699090.341960:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699090.341963:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699090/real 1787699090] req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699090.341966:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699090.341979:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699090.341988:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699090.341990:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699090.342014:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699090.342018:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699090.342021:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:3.0:1787699090.342021:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699090.342027:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6991 00000100:00080000:1.0:1787699090.342032:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699090.342036:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.342040:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00080000:3.0:1787699090.342040:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000020:00000040:1.0:1787699090.342044:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699090.342047:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00080000:3.0:1787699090.342049:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:1.0:1787699090.342051:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699090.342055:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699090.342059:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00000040:3.0:1787699090.342060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699090.342062:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699090.342068:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699090.342072:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699090.342079:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126420096:192.168.201.136@tcp:250:kworker.0 00000100:00100000:1.0:1787699090.342080:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699090.342094:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126420096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699090.342104:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699090.342117:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7001) 00000020:00000040:3.0:1787699090.342136:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699090.342155:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342164:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699090.342180:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342194:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699090.342207:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699090.342214:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342225:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699090.342231:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699090/real 1787699090] req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342243:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699090.342260:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699090.342263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699090.342267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699090.342292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342300:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699090.342313:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342320:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699090.342332:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699090.342338:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342347:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699090.342353:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699090/real 1787699090] req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342368:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342376:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699090.342384:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699090.342387:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699090.342390:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342397:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699090.342403:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342446:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699090.342462:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342476:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699090.342489:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699090.342496:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342505:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699090.342511:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699090/real 1787699090] req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342522:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699090.342534:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699090.342536:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699090.342539:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342543:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699090.342547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342551:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699090.342564:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342569:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699090.342577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699090.342583:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699115 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342592:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699090.342598:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699090/real 1787699090] req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699090.342617:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699090.342624:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699090.342627:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699090.342630:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699090.342635:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699090.342639:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126420224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699090.342644:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699090.342648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126420352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699090.342661:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699090.342665:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126420480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699090.342669:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699090.342672:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126420608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699115 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699090.342676:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699090.853545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699090.853546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699090.853566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699090.981579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699091.877532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699091.877532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699091.877532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699092.005639:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699092.261558:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301665647, at 4301665664 00000100:00000040:3.0:1787699092.261558:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301665647, at 4301665664 00000100:00100000:0.0:1787699092.901552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699092.901552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699092.901554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699093.029633:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699093.925534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699093.925535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699093.925535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699094.053638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699094.949530:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699094.949530:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699094.949530:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699095.077534:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699095.461575:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699095.461589:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699095.461595:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699095.461608:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 6996 00000100:00080000:2.0:1787699095.461622:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699095.461626:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461631:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461635:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699095.461641:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699095.461644:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461646:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461648:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699095.461652:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461672:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699095.461714:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699095.461732:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699095.461759:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699095.461763:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699095.461765:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699095.461770:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6996 00000100:00080000:2.0:1787699095.461774:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699095.461777:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461781:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461783:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699095.461785:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699095.461787:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461790:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461791:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699095.461794:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699095.461804:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:3.0:1787699095.461809:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699095.461811:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699095.461827:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699095.461829:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126420736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787699095.461844:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699095.461851:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699095.461853:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699095.461855:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699095.461858:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6996 00000100:00080000:2.0:1787699095.461862:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699095.461862:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699095.461868:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461873:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461877:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699095.461881:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699095.461881:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461888:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461890:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699095.461892:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699095.461896:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699095.461899:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699095.461905:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699095.461910:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699095.461916:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699095.461918:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699095.461919:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699095.461922:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6996 00000100:00080000:2.0:1787699095.461924:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699095.461925:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461929:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:3.0:1787699095.461931:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461932:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699095.461936:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699095.461939:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.461942:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.461945:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699095.461948:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699095.461965:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:3.0:1787699095.461970:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699095.461973:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699095.461988:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699095.461997:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699095.462004:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699095.462006:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699095.462008:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699095.462010:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 6996 00000100:00080000:2.0:1787699095.462014:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787699095.462017:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699095/real 1787699095] req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699095.462019:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.462025:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.462028:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699095.462032:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699095.462035:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699095.462036:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699095.462038:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699095.462040:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699095.462044:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699095.462044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699095.462049:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699095.462055:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699095.462071:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699095.462082:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7006) 00000100:00080000:3.0:1787699095.462089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699095.462092:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699095.462095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462100:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126420736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699095.462108:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126420736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699095.462115:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699095.462128:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699095.462145:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462150:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699095.462158:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699095.462163:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462170:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699095.462173:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699095/real 1787699095] req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462193:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462201:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699095.462208:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699095.462212:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699095.462216:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699095.462230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462238:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699095.462249:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462254:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699095.462262:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699095.462266:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462272:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699095.462276:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699095/real 1787699095] req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462304:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462310:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699095.462315:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699095.462316:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699095.462318:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462323:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126420992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699095.462326:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462331:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699095.462339:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462343:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699095.462350:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699095.462354:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462363:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699095.462368:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699095/real 1787699095] req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699095.462407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699095.462410:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699095.462414:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699095.462449:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462454:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699095.462463:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462467:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699095.462475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699095.462480:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699120 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462486:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699095.462489:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699095/real 1787699095] req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462495:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699095.462505:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699095.462509:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699095.462511:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699095.462513:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699095.462517:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699095.462520:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126420864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699095.462525:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699095.462528:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126420992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699095.462532:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699095.462535:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126421120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699095.462539:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699095.462542:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126421248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699120 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699095.462546:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699095.973538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699095.973538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699095.973541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699096.101600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699096.997524:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699096.997524:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699096.997524:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699097.125644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699097.381500:0:88496:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301670767, at 4301670784 00000100:00000040:2.0:1787699097.381501:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301670767, at 4301670784 00000100:00100000:2.0:1787699098.021549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699098.021554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699098.021579:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699098.149565:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699099.045540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699099.045544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699099.045593:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699099.173592:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699100.069546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699100.069546:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699100.069551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699100.197627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699100.581582:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699100.581626:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699100.581632:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699100.581645:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7001 00000100:00080000:0.0:1787699100.581658:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699100.581662:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.581667:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.581671:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699100.581677:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699100.581681:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.581683:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.581688:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699100.581691:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699100.581710:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699100.581742:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699100.581759:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699100.581789:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699100.581793:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699100.581795:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699100.581799:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7001 00000100:00080000:0.0:1787699100.581804:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699100.581807:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.581809:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.581812:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699100.581815:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699100.581818:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.581821:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.581823:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699100.581826:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699100.581837:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699100.581842:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699100.581852:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699100.581863:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699100.581878:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699100.581882:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699100.581884:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699100.581888:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7001 00000100:00080000:0.0:1787699100.581892:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699100.581895:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.581898:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.581901:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699100.581904:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699100.581907:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.581912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.581915:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:3.0:1787699100.581915:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699100.581918:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699100.581924:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699100.581932:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699100.581941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126421376:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787699100.581946:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699100.581957:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699100.581967:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699100.581970:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699100.581973:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699100.581981:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7001 00000100:00000040:3.0:1787699100.581981:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699100.581986:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699100.581993:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.582001:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699100.582002:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:0.0:1787699100.582006:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699100.582010:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699100.582013:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.582016:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.582019:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699100.582022:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699100.582028:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699100.582034:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699100.582045:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699100.582054:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699100.582063:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699100.582066:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699100.582068:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699100.582072:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7001 00000100:00080000:0.0:1787699100.582076:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699100.582078:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.582081:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.582083:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699100.582085:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699100.582088:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699100.582091:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699100.582093:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699100.582097:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699100.582101:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699100.582105:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699100.582114:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699100.582123:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699100.582133:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7011) 00000800:00000100:3.0:1787699100.582190:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699100.582210:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582239:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699100/real 1787699100] req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582252:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582260:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699100.582269:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699100.582273:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699100.582278:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582286:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126421376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699100.582294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582303:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699100.582331:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582336:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699100.582345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699100.582360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582374:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699100.582380:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699100/real 1787699100] req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699100.582407:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699100.582411:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699100.582415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699100.582451:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126421376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699100.582462:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699100.582472:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126421504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699100.582489:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699100.582499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699100.582515:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582522:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699100.582534:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699100.582541:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582550:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699100.582556:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699100/real 1787699100] req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582568:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582586:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699100.582605:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699100.582608:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699100.582612:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699100.582633:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582642:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699100.582654:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582661:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699100.582673:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699100.582680:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582688:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699100.582693:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699100/real 1787699100] req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582703:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582711:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699100.582718:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699100.582722:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699100.582725:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582732:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699100.582738:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582750:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699100.582759:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582764:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699100.582771:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699100.582775:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699125 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582781:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699100.582785:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699100/real 1787699100] req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582791:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699100.582796:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699100.582800:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699100.582802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699100.582804:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699100.582808:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126421888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699100.582811:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126421632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699100.582816:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699100.582825:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126421760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699100.582830:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699100.582834:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126421888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699125 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699100.582841:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699101.093584:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699101.093596:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699101.093604:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699101.221647:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699102.117532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699102.117532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699102.117532:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699102.245606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699102.501528:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301675887, at 4301675904 00000100:00000040:3.0:1787699102.501528:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301675887, at 4301675904 00000100:00100000:0.0:1787699103.141566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699103.141566:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699103.141566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699103.269666:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699104.165565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699104.165565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699104.165565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699104.293626:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699105.189543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699105.189586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699105.189841:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699105.317588:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699105.701555:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699105.701568:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699105.701574:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699105.701591:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7006 00000100:00080000:2.0:1787699105.701604:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699105.701608:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701612:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701616:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699105.701622:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699105.701626:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701628:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701632:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699105.701635:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701653:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699105.701683:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699105.701695:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699105.701718:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699105.701721:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699105.701722:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699105.701725:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7006 00000100:00080000:2.0:1787699105.701728:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699105.701729:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701735:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701737:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699105.701739:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699105.701741:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701746:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701748:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699105.701750:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699105.701759:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699105.701762:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699105.701771:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699105.701778:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699105.701781:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699105.701804:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699105.701807:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699105.701808:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699105.701810:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7006 00000100:00080000:2.0:1787699105.701812:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787699105.701815:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126422016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787699105.701816:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701820:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701824:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699105.701825:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699105.701827:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701829:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701830:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699105.701831:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699105.701834:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699105.701836:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699105.701842:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699105.701847:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699105.701852:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699105.701854:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699105.701856:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:3.0:1787699105.701860:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699105.701861:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7006 00000100:00080000:2.0:1787699105.701865:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699105.701870:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701875:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701879:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699105.701881:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699105.701886:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699105.701886:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701890:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701891:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699105.701893:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699105.701896:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699105.701899:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699105.701903:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699105.701909:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699105.701913:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699105.701915:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699105.701917:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699105.701919:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7006 00000100:00080000:2.0:1787699105.701922:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699105.701924:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701925:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701926:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699105.701930:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699105.701933:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000800:00000100:3.0:1787699105.701938:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699105.701939:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699105.701942:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699105.701945:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699105.701948:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699105.701952:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699105.701956:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699105.701962:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699105.701973:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7016) 00000100:00000040:3.0:1787699105.701978:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699105/real 1787699105] req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702046:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702057:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699105.702084:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699105.702088:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699105.702093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702102:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126422016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699105.702109:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126422016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699105.702121:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699105.702139:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702149:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699105.702165:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702173:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699105.702186:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699105.702194:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702204:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699105.702211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699105/real 1787699105] req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702223:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702238:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699105.702245:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699105.702248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699105.702252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699105.702266:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699105.702288:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702294:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699105.702303:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699105.702308:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702314:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699105.702318:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699105/real 1787699105] req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702324:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699105.702336:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699105.702338:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699105.702340:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702345:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699105.702349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702356:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699105.702368:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702376:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699105.702386:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699105.702393:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702402:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699105.702407:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699105/real 1787699105] req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702417:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702442:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699105.702447:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699105.702450:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699105.702452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699105.702460:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702465:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699105.702472:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702478:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699105.702488:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699105.702495:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699130 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702503:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699105.702520:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699105/real 1787699105] req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702533:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699105.702541:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699105.702549:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699105.702552:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699105.702556:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699105.702565:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699105.702570:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126422144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699105.702578:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699105.702584:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126422272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699105.702589:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699105.702592:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126422400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699105.702596:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699105.702599:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126422528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699130 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699105.702602:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699106.213554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699106.213555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699106.213565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699106.341638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699107.237548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699107.237548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699107.237548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699107.366724:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699107.621548:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301681007, at 4301681024 00000100:00000040:2.0:1787699107.621643:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301681007, at 4301681024 00000100:00100000:0.0:1787699108.261551:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699108.261551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699108.261561:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699108.391064:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699109.285537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699109.285537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699109.285648:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699109.413637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699110.309558:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699110.309558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699110.309613:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699110.437600:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699110.821657:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699110.821669:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699110.821674:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699110.821688:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7011 00000100:00080000:3.0:1787699110.821701:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699110.821705:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.821710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.821714:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699110.821719:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699110.821722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.821726:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.821729:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699110.821732:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699110.821750:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699110.821786:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699110.821802:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699110.821831:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699110.821835:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699110.821837:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699110.821842:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7011 00000100:00080000:3.0:1787699110.821846:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699110.821849:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.821851:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.821853:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699110.821856:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699110.821867:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.821870:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.821872:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699110.821875:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699110.821887:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699110.821892:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699110.821901:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699110.821911:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699110.821919:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699110.821923:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699110.821924:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699110.821928:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7011 00000100:00080000:3.0:1787699110.821938:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699110.821940:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.821943:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.821945:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699110.821947:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699110.821950:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.821952:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.821954:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699110.821957:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699110.821961:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699110.821966:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699110.821974:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699110.821982:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699110.821990:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699110.821993:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699110.821995:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699110.821999:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7011 00000100:00080000:3.0:1787699110.822001:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699110.822004:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.822007:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.822009:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699110.822011:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699110.822014:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.822023:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.822027:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699110.822030:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699110.822034:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699110.822038:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699110.822045:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699110.822053:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699110.822060:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699110.822066:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699110.822067:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699110.822071:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7011 00000100:00080000:3.0:1787699110.822073:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699110.822076:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.822078:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.822080:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699110.822082:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699110.822084:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699110.822087:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699110.822089:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699110.822091:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699110.822095:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699110.822098:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699110.822105:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699110.822113:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699110.822123:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7021) 00000100:00000040:0.0:1787699110.822198:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822214:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126422656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699110.822246:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822253:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699110.822296:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699110.822315:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822354:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699110/real 1787699110] req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822370:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699110.822389:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699110.822393:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699110.822397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822405:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126422656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699110.822411:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822467:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699110.822482:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822490:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699110.822500:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699110.822507:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822517:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699110.822522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699110/real 1787699110] req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822532:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822540:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699110.822547:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699110.822550:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699110.822562:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822569:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699110.822574:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822581:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699110.822592:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822599:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699110.822608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699110.822615:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822622:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699110.822627:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699110/real 1787699110] req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822643:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822651:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699110.822658:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699110.822662:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699110.822665:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822673:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126422912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699110.822677:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822685:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699110.822697:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822703:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699110.822713:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699110.822718:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822726:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699110.822731:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699110/real 1787699110] req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822746:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699110.822761:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699110.822764:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699110.822768:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822775:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699110.822780:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822788:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699110.822798:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822805:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699110.822815:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699110.822821:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699135 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822830:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699110.822835:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699110/real 1787699110] req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822845:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699110.822854:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699110.822861:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699110.822864:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699110.822868:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699110.822874:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699110.822880:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126422656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699110.822890:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699110.822899:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126422784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699110.822906:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699110.822911:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126422912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699110.822917:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699110.822922:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126423040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699110.822929:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699110.822933:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126423168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699135 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699110.822940:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699111.333560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699111.333560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699111.333560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699111.461624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699112.357559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699112.357561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699112.357652:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699112.485962:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699112.741553:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301686127, at 4301686144 00000100:00000040:3.0:1787699112.741553:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301686127, at 4301686144 00000100:00100000:0.0:1787699113.381572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699113.381572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699113.381573:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699113.509645:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699114.405544:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699114.405544:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699114.405564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699114.533635:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699115.429541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699115.429541:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699115.429651:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699115.557607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699115.941546:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699115.941559:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699115.941565:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699115.941578:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7016 00000100:00080000:0.0:1787699115.941591:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699115.941594:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941597:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941601:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699115.941606:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699115.941610:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941613:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941616:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699115.941619:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941636:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699115.941669:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699115.941686:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699115.941711:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699115.941714:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699115.941716:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699115.941719:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7016 00000100:00080000:0.0:1787699115.941722:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699115.941725:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941727:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941729:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699115.941732:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699115.941735:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941737:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941739:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699115.941741:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699115.941750:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:3.0:1787699115.941751:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699115.941757:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699115.941772:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126423296:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787699115.941773:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699115.941788:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699115.941798:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699115.941801:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699115.941802:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699115.941807:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7016 00000100:00000040:3.0:1787699115.941808:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699115.941811:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699115.941816:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941821:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941825:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699115.941828:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699115.941832:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941833:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941837:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941839:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699115.941841:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699115.941845:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699115.941849:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699115.941856:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699115.941861:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699115.941867:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699115.941870:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699115.941871:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699115.941877:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7016 00000800:00000100:3.0:1787699115.941879:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699115.941880:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699115.941884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941887:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699115.941889:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699115.941890:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941892:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941893:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699115.941895:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699115.941899:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:3.0:1787699115.941903:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699115.941904:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699115.941915:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699115.941921:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699115.941927:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699115.941929:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699115.941930:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699115.941933:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7016 00000100:00080000:0.0:1787699115.941935:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699115.941937:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941939:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941940:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699115.941942:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00100000:3.0:1787699115.941945:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699115/real 1787699115] req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699115.941946:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699115.941951:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699115.941959:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699115.941961:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699115.941965:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1787699115.941969:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699115.941970:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699115.941980:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699115.942005:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699115.942007:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699115.942026:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:0.0:1787699115.942031:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7027) 00000100:00080000:3.0:1787699115.942032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699115.942040:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942048:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126423296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699115.942057:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126423296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699115.942068:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699115.942083:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942092:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699115.942104:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942111:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699115.942126:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699115.942133:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942142:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699115.942148:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699115/real 1787699115] req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942165:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942173:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699115.942181:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699115.942184:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699115.942188:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942196:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699115.942202:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699115.942220:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942227:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699115.942235:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699115.942241:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942249:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699115.942254:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699115/real 1787699115] req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942271:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942280:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699115.942287:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699115.942290:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699115.942294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942302:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699115.942306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942315:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699115.942326:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942347:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699115.942357:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699115.942363:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942372:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699115.942377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699115/real 1787699115] req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942395:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942404:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699115.942411:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699115.942414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699115.942434:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942443:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699115.942448:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942457:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699115.942467:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942475:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699115.942484:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699115.942490:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699140 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942499:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699115.942504:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699115/real 1787699115] req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942515:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699115.942524:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699115.942531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699115.942534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699115.942538:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699115.942545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126423808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699115.942551:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126423424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699115.942558:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699115.942564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126423552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699115.942571:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699115.942575:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126423680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699115.942583:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699115.942588:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126423808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699140 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699115.942595:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699116.453526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699116.453526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699116.453528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699116.581637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699117.477533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699117.477636:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699117.478258:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699117.605591:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699117.861540:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301691247, at 4301691264 00000100:00000040:3.0:1787699117.861540:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301691247, at 4301691264 00000100:00100000:0.0:1787699118.501518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699118.501518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699118.501518:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699118.629623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699119.525576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699119.525576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699119.525947:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699119.653654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699120.549536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699120.549536:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699120.550481:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699120.678174:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699121.061515:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699121.061525:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699121.061529:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699121.061546:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7022 00000100:00080000:0.0:1787699121.061555:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699121.061557:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061560:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061562:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699121.061566:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699121.061569:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061570:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061572:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699121.061574:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061586:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699121.061610:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699121.061630:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699121.061654:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699121.061662:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699121.061664:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699121.061668:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7022 00000100:00080000:0.0:1787699121.061671:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699121.061672:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061674:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061675:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699121.061678:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699121.061680:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061682:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061687:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699121.061689:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699121.061696:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699121.061700:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699121.061707:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699121.061718:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699121.061719:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699121.061731:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699121.061738:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699121.061740:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699121.061742:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7022 00000100:00080000:0.0:1787699121.061745:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787699121.061746:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126423936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699121.061751:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061754:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061755:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699121.061757:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699121.061758:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061760:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061761:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699121.061762:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699121.061765:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699121.061767:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699121.061773:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699121.061778:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699121.061787:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:3.0:1787699121.061788:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699121.061801:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699121.061803:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699121.061807:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699121.061812:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7022 00000100:00080000:0.0:1787699121.061817:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699121.061820:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061822:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061825:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699121.061827:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699121.061830:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061833:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061835:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699121.061838:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699121.061844:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699121.061848:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699121.061859:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699121.061869:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699121.061876:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699121.061881:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787699121.061885:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424064:192.168.201.136@tcp:38:kworker.0 00000100:00080000:0.0:1787699121.061886:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699121.061890:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699121.061894:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7022 00000100:00080000:0.0:1787699121.061896:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699121.061900:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699121.061901:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699121.061905:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061910:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699121.061913:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699121.061916:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699121.061918:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699121.061921:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699121.061927:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:3.0:1787699121.061928:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:0.0:1787699121.061937:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699121.061942:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699121.061945:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699121.061955:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699121.061956:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699121.061988:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699121.061990:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699121.062001:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7032) 00000100:00000040:3.0:1787699121.062007:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699121.062013:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699121.062023:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787699121.062038:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699121.062052:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699121.062066:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699121.062073:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699121.062083:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787699121.062089:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699121.062097:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699121.062109:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699121.062114:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699121.062125:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:1.0:1787699121.573549:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699121.573549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699121.573616:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699121.701582:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699122.597528:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699122.597528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699122.597531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699122.725638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699122.981546:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301696367, at 4301696384 00000100:00000040:3.0:1787699122.981552:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301696367, at 4301696384 00000100:00100000:0.0:1787699123.621559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699123.621559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699123.621764:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699123.749654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1787699124.133818:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:1.0:1787699124.133877:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787699124.133897:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699124.133945:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699124.133965:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699124.133975:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699124.133990:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699124.133998:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699124.134008:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699124.134016:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787699124.134025:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:1.0:1787699124.134033:0:2387:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699146 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699124] req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134604:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699124.134616:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699124.134628:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699124.134633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699124.134638:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134646:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424320:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787699124.134657:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699124] req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134669:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699124.134676:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699124.134682:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699124.134686:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699124.134690:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424448:192.168.201.136@tcp:8:kworker.0 00000100:00100000:3.0:1787699124.134702:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699124] req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134714:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699124.134723:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699124.134731:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699124.134735:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699124.134738:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424064:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787699124.134758:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699124] req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134770:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699124.134778:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699124.134787:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699124.134790:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699124.134794:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134802:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424192:192.168.201.136@tcp:38:kworker.0 00000100:00100000:3.0:1787699124.134808:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699121/real 1787699124] req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134819:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699124.134826:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699124.134832:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699124.134835:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699124.134840:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699124.134846:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126423936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699124.134852:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126424320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699124.134866:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699124.134878:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126424448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699124.134885:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699124.134891:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126424064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699124.134910:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699124.134916:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126424192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699124.134922:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699124.134927:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126423936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699146 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699124.134933:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699124.645843:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699124.647088:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699124.647747:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699124.773638:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699125.669576:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699125.669593:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699125.669640:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699125.797971:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699126.181577:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699126.181590:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699126.181595:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699126.181608:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7027 00000100:00080000:3.0:1787699126.181620:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699126.181624:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.181629:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.181633:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699126.181639:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699126.181642:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.181645:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.181648:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699126.181652:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699126.181665:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699126.181696:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699126.181712:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699126.181740:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699126.181743:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699126.181745:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699126.181749:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7027 00000100:00080000:3.0:1787699126.181753:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699126.181755:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.181758:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.181760:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699126.181763:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699126.181765:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.181767:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.181769:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699126.181772:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699126.181782:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699126.181787:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699126.181798:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699126.181809:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.181816:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699126.181821:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699126.181829:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699126.181832:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699126.181839:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7027 00000100:00080000:3.0:1787699126.181842:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699126.181847:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126424576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699126.181847:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.181853:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.181857:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699126.181859:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699126.181862:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.181864:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.181866:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699126.181868:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699126.181873:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699126.181878:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699126.181885:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699126.181890:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699126.181904:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.181913:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699126.181925:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699126.181929:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699126.181932:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699126.181937:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7027 00000100:00080000:3.0:1787699126.181941:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699126.181944:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.181947:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000800:00000100:0.0:1787699126.181967:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699126.181985:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182024:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699126/real 1787699126] req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182044:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182054:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699126.182064:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699126.182069:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699126.182073:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182081:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126424576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699126.182089:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126424576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699126.182099:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699126.182118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182128:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699126.182144:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182155:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787699126.182354:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699126.182359:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699126.182364:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.182366:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.182369:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699126.182371:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699126.182377:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699126.182381:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699126.182391:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699126.182402:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699126.182412:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787699126.182439:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699126.182450:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699126.182457:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699126.182461:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699126.182467:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7027 00000100:00000040:0.0:1787699126.182471:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787699126.182472:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699126.182478:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787699126.182482:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699126/real 1787699126] req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699126.182484:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699126.182490:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699126.182494:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699126.182498:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699126.182501:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00100000:0.0:1787699126.182505:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699126.182505:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699126.182512:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699126.182521:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787699126.182525:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699126.182528:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787699126.182539:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787699126.182541:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699126.182548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699126.182557:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699126.182562:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182578:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699126.182583:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7037) 00000100:00000040:0.0:1787699126.182588:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182600:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699126.182615:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182623:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699126.182637:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699126.182646:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182656:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699126.182662:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699126/real 1787699126] req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182686:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182693:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699126.182700:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699126.182703:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699126.182707:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182713:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699126.182718:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126424704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699126.182725:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699126.182732:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126424832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699126.182739:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699126.182748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699126.182775:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182782:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699126.182793:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699126.182799:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182808:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699126.182813:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699126/real 1787699126] req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182822:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182830:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699126.182836:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699126.182838:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699126.182842:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182848:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126424960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699126.182852:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182859:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699126.182869:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182875:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699126.182885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699126.182891:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699151 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182899:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699126.182903:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699126/real 1787699126] req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182913:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699126.182921:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699126.182928:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699126.182931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699126.182934:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699126.182942:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699126.182948:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126424960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699126.182955:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699126.182960:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126425088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699151 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699126.182972:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699126.693558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699126.693558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699126.694115:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699126.821621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699127.717544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699127.717566:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699127.717677:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699127.845755:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699128.101542:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301701487, at 4301701504 00000100:00000040:3.0:1787699128.101542:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301701487, at 4301701504 00000100:00100000:1.0:1787699128.741514:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699128.741514:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699128.741647:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699128.869712:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699129.765528:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699129.765528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699129.765554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699129.893641:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699130.789587:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699130.789630:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699130.789761:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699130.917660:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699131.301522:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699131.301536:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699131.301541:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699131.301553:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7032 00000100:00080000:3.0:1787699131.301565:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699131.301568:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301572:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301575:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699131.301580:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699131.301582:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301594:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301597:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699131.301601:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301619:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699131.301652:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699131.301665:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699131.301687:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699131.301690:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699131.301696:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699131.301701:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7032 00000100:00080000:3.0:1787699131.301704:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699131.301707:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301710:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301712:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699131.301715:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699131.301718:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301720:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301723:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699131.301725:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699131.301734:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699131.301736:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699131.301741:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699131.301749:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126425216:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787699131.301750:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699131.301759:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699131.301779:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787699131.301782:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699131.301783:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699131.301785:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699131.301789:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7032 00000100:00000040:0.0:1787699131.301791:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699131.301792:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699131.301795:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301798:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301799:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699131.301801:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699131.301804:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301808:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699131.301810:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699131.301814:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699131.301818:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699131.301825:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699131.301833:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787699131.301834:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699131.301840:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699131.301843:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699131.301844:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699131.301848:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7032 00000100:00080000:3.0:1787699131.301850:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699131.301853:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699131.301853:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301856:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301858:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699131.301860:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699131.301863:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301865:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301867:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699131.301869:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699131.301873:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699131.301877:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699131.301883:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699131.301889:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699131/real 1787699131] req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699131.301891:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699131.301899:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699131.301902:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699131.301903:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699131.301907:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699131.301907:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7032 00000100:00080000:3.0:1787699131.301910:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699131.301912:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699131.301917:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699131.301917:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699131.301919:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699131.301922:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699131.301924:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699131.301926:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699131.301927:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699131.301928:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:0.0:1787699131.301931:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:3.0:1787699131.301932:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787699131.301935:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699131.301936:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699131.301942:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699131.301944:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126425216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699131.301951:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126425216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699131.301951:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699131.301961:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7042) 00000020:00000040:0.0:1787699131.301968:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699131.301987:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.301995:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699131.302009:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302017:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699131.302029:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699131.302036:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302047:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699131.302052:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699131/real 1787699131] req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302063:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302087:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699131.302096:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699131.302099:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699131.302104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302118:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699131.302126:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302135:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699131.302146:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302154:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699131.302163:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699131.302175:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302185:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699131.302190:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699131/real 1787699131] req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302201:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302208:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699131.302216:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699131.302218:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699131.302222:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302230:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699131.302235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302244:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699131.302256:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302263:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699131.302272:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699131.302278:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302294:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699131.302303:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699131/real 1787699131] req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302322:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699131.302329:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699131.302331:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699131.302334:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699131.302349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302357:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699131.302368:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302375:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699131.302383:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699131.302388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699156 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302396:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699131.302401:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699131/real 1787699131] req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302411:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699131.302453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699131.302461:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699131.302464:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699131.302468:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699131.302476:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699131.302481:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126425344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699131.302489:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699131.302494:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126425472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699131.302515:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699131.302520:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126425600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699131.302527:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699131.302532:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126425728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699156 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699131.302538:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699131.813589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699131.813589:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699131.814445:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699131.942234:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699132.837564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699132.837565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699132.837575:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699132.965668:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699133.221645:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301706607, at 4301706624 00000100:00000040:2.0:1787699133.222795:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301706607, at 4301706625 00000100:00100000:1.0:1787699133.861532:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699133.861554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699133.861588:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699133.989644:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699134.885563:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699134.885563:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699134.885568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699135.013683:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699135.909537:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699135.909537:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699135.909537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699136.037628:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699136.421582:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699136.421596:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699136.421602:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699136.421615:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7037 00000100:00080000:3.0:1787699136.421630:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699136.421633:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421638:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.421642:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699136.421648:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699136.421651:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421654:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.421657:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699136.421660:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699136.421682:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699136.421714:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699136.421731:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699136.421756:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699136.421760:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699136.421761:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699136.421765:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7037 00000100:00080000:3.0:1787699136.421769:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699136.421772:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421774:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.421776:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699136.421780:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699136.421783:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421786:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.421788:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699136.421798:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699136.421809:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787699136.421812:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699136.421821:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699136.421835:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699136.421837:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126425856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699136.421853:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699136.421863:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699136.421867:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699136.421869:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699136.421872:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7037 00000100:00080000:3.0:1787699136.421876:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699136.421879:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421884:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699136.421885:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699136.421888:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699136.421892:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699136.421900:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421907:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699136.421908:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:3.0:1787699136.421912:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699136.421916:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699136.421921:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699136.421926:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699136.421938:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699136.421948:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699136.421957:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699136.421961:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699136.421963:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699136.421969:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7037 00000100:00080000:3.0:1787699136.421972:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699136.421978:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421978:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.421984:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.422017:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000100:00000040:0.0:1787699136.422020:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699136.422022:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699136.422030:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.422036:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.422040:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699136.422043:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699136.422049:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699136.422054:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699136.422064:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699136.422074:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699136/real 1787699136] req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699136.422076:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699136.422105:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699136.422109:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699136.422111:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699136.422116:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7037 00000100:00100000:0.0:1787699136.422120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699136.422122:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699136.422130:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.422137:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.422140:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00000040:0.0:1787699136.422143:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699136.422144:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699136.422151:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699136.422158:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699136.422162:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699136.422163:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699136.422166:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:0.0:1787699136.422172:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:3.0:1787699136.422174:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787699136.422179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699136.422182:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699136.422194:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126425856:192.168.201.136@tcp:250:kworker.0 00000100:00100000:3.0:1787699136.422196:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699136.422209:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126425856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699136.422214:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699136.422233:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699136.422234:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7047) 00000100:00000040:0.0:1787699136.422258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422268:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699136.422285:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422292:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699136.422307:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699136.422314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422327:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699136.422333:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699136/real 1787699136] req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422345:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422353:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699136.422379:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699136.422383:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699136.422388:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422395:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126425984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699136.422401:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422409:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699136.422451:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422459:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699136.422475:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699136.422482:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422492:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699136.422497:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699136/real 1787699136] req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422508:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422516:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699136.422525:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699136.422528:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699136.422532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422547:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699136.422555:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422563:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699136.422575:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422582:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699136.422592:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699136.422598:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422607:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699136.422612:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699136/real 1787699136] req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422628:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699136.422634:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699136.422636:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699136.422639:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422647:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699136.422661:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422668:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699136.422679:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422686:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699136.422695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699136.422701:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699161 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422709:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699136.422714:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699136/real 1787699136] req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422723:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699136.422731:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699136.422737:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699136.422740:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699136.422744:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699136.422751:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699136.422757:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126425984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699136.422765:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699136.422771:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126426112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699136.422778:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699136.422783:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126426240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699136.422789:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699136.422794:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126426368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699161 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699136.422800:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699136.933489:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699136.933489:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699136.933535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699137.062168:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699137.957556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699137.957556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699137.957572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699138.085640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699138.341563:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301711727, at 4301711744 00000100:00000040:3.0:1787699138.341563:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301711727, at 4301711744 00000100:00100000:2.0:1787699138.981583:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699138.981583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699138.981584:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699139.109602:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699140.005589:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699140.005589:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699140.005986:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699140.133624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699141.029519:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699141.029519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699141.029572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699141.157585:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699141.541527:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699141.541537:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699141.541541:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699141.541553:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7042 00000100:00080000:2.0:1787699141.541567:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699141.541570:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541574:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541577:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699141.541582:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699141.541584:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541586:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541589:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699141.541591:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541605:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699141.541632:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699141.541645:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699141.541663:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699141.541667:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699141.541670:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699141.541675:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7042 00000100:00080000:2.0:1787699141.541679:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699141.541682:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541686:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541688:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699141.541691:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699141.541695:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541698:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541701:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699141.541703:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699141.541712:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699141.541715:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699141.541724:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699141.541736:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699141.541738:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126426496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:2.0:1787699141.541755:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699141.541761:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699141.541764:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699141.541765:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699141.541768:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7042 00000100:00080000:2.0:1787699141.541772:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699141.541775:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699141.541779:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541786:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541790:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699141.541793:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699141.541797:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541800:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541803:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541804:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699141.541806:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699141.541811:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699141.541817:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699141.541826:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699141.541836:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787699141.541844:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:2.0:1787699141.541851:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699141.541856:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699141.541858:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699141.541863:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7042 00000100:00080000:2.0:1787699141.541867:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699141.541868:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699141.541871:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541877:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541882:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699141.541885:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699141.541889:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541891:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541892:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699141.541894:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699141.541913:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699141.541918:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699141.541926:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699141.541933:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699141.541939:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699141.541942:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699141.541943:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699141.541946:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7042 00000100:00080000:2.0:1787699141.541949:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787699141.541952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699141/real 1787699141] req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699141.541953:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541959:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541962:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699141.541966:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699141.541971:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699141.541974:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699141.541975:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699141.541977:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699141.541981:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699141.541985:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699141.541986:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699141.541996:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699141.542003:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699141.542005:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699141.542030:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787699141.542031:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7052) 00000100:00080000:0.0:1787699141.542038:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699141.542044:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542054:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126426496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699141.542062:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126426496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699141.542073:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699141.542095:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542104:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699141.542116:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542125:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699141.542136:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699141.542143:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542154:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699141.542159:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699141/real 1787699141] req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542175:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542184:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699141.542191:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699141.542194:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699141.542198:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542205:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699141.542210:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542217:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699141.542229:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542236:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699141.542246:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699141.542252:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542261:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699141.542266:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699141/real 1787699141] req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542283:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542292:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699141.542300:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699141.542303:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699141.542307:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542315:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699141.542320:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542327:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699141.542338:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542346:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699141.542354:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699141.542361:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542369:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699141.542374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699141/real 1787699141] req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542391:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542399:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699141.542406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699141.542409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699141.542413:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126426880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699141.542451:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542459:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699141.542470:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542478:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699141.542487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699141.542494:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699166 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542502:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699141.542508:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699141/real 1787699141] req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542518:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699141.542527:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699141.542535:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699141.542538:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699141.542542:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699141.542549:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699141.542554:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126426624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699141.542562:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699141.542568:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126426752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699141.542575:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699141.542581:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126426880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699141.542587:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699141.542591:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126427008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699166 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699141.542598:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699142.053577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699142.053577:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699142.053577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699142.181668:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699143.077582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699143.077583:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699143.077586:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699143.205622:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699143.461739:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301716847, at 4301716864 00000100:00000040:2.0:1787699143.461819:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301716847, at 4301716864 00000100:00100000:1.0:1787699144.101537:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699144.101554:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699144.101559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699144.229654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699145.125548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699145.125548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699145.125549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699145.253606:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699146.149493:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699146.149511:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699146.149542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699146.277550:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699146.661528:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699146.661536:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699146.661539:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699146.661547:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7047 00000100:00080000:3.0:1787699146.661556:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699146.661558:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661561:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661563:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699146.661567:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699146.661569:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661571:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661574:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699146.661577:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661591:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699146.661630:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699146.661647:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699146.661668:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699146.661672:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699146.661674:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699146.661678:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7047 00000100:00080000:3.0:1787699146.661682:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699146.661684:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661687:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661690:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699146.661693:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699146.661697:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661701:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661704:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699146.661706:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699146.661716:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699146.661721:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699146.661730:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699146.661737:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699146.661779:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126427136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699146.661781:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699146.661798:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699146.661802:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699146.661804:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699146.661809:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7047 00000100:00080000:3.0:1787699146.661823:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699146.661826:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661828:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661830:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699146.661833:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699146.661835:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661841:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699146.661842:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699146.661848:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699146.661863:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699146.661867:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699146.661879:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699146.661886:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:3.0:1787699146.661892:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699146.661901:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699146.661906:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699146.661908:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699146.661909:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699146.661912:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7047 00000100:00080000:3.0:1787699146.661916:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699146.661917:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661919:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661920:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699146.661921:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699146.661923:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661925:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.661927:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699146.661928:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699146.661932:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699146.661935:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699146.661940:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699146.661946:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699146.661961:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699146.661968:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699146.661971:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699146.661974:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787699146.661977:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699146.661980:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7047 00000100:00080000:3.0:1787699146.661991:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699146.662001:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.662015:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.662018:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699146.662020:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699146.662024:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699146.662027:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699146.662028:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699146.662029:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699146.662033:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699146.662036:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699146.662048:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699146.662056:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699146/real 1787699146] req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699146.662061:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699146.662137:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7057) 00000100:00100000:0.0:1787699146.662147:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699146.662164:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699146.662167:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699146.662170:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662176:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126427136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699146.662180:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126427136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699146.662188:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699146.662203:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662210:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699146.662222:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662227:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699146.662243:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699146.662248:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662256:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699146.662260:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699146/real 1787699146] req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662267:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662272:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699146.662277:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699146.662279:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699146.662281:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662285:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699146.662288:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662299:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699146.662306:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662311:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699146.662317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699146.662321:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662326:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699146.662329:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699146/real 1787699146] req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662336:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662341:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699146.662345:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699146.662346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699146.662348:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662352:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699146.662356:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662361:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699146.662370:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662374:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699146.662379:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699146.662383:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662388:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699146.662391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699146/real 1787699146] req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662403:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699146.662413:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699146.662415:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699146.662461:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699146.662470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662475:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699146.662483:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662491:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699146.662497:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699146.662502:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699171 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662511:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699146.662516:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699146/real 1787699146] req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662526:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699146.662535:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699146.662542:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699146.662545:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699146.662549:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699146.662557:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699146.662560:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126427264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699146.662566:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699146.662570:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126427392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699146.662575:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699146.662578:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126427520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699146.662583:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699146.662586:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126427648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699171 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699146.662590:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699147.173533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699147.173533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699147.173546:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699147.301543:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699148.197611:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699148.197621:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699148.197622:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699148.325609:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699148.581516:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301721967, at 4301721984 00000100:00000040:3.0:1787699148.581518:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301721967, at 4301721984 00000100:00100000:0.0:1787699149.221531:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699149.221531:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699149.221563:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699149.349654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699150.245515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699150.245515:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699150.245519:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699150.373579:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699151.269509:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699151.269509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699151.269539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699151.397596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699151.781599:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699151.781612:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699151.781617:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699151.781631:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7052 00000100:00080000:2.0:1787699151.781643:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699151.781647:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.781651:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.781655:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699151.781660:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699151.781673:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.781676:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.781680:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699151.781683:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699151.781701:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699151.781746:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699151.781764:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699151.781793:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699151.781798:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699151.781799:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699151.781804:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7052 00000100:00080000:2.0:1787699151.781808:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699151.781811:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.781813:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.781815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699151.781818:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699151.781821:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.781832:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.781834:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699151.781837:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699151.781848:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699151.781854:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699151.781865:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699151.781878:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699151.781879:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699151.781908:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699151.781912:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1787699151.781917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126427776:192.168.201.136@tcp:250:kworker.0 00000100:00080000:2.0:1787699151.781917:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699151.781923:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7052 00000100:00080000:2.0:1787699151.781938:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699151.781945:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.781948:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.781951:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699151.781954:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699151.781958:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699151.781959:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.781965:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.781969:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699151.781975:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699151.781980:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000008:00100000:2.0:1787699151.781981:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699151.781987:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699151.781998:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699151.782007:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699151.782016:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699151.782019:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699151.782021:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699151.782025:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7052 00000100:00080000:2.0:1787699151.782028:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787699151.782030:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699151.782034:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.782038:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.782040:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699151.782042:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699151.782046:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.782049:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.782052:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:0.0:1787699151.782056:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699151.782057:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699151.782065:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699151.782072:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699151.782083:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699151.782092:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699151.782100:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699151.782104:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699151.782106:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699151.782110:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7052 00000100:00100000:0.0:1787699151.782114:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699151/real 1787699151] req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699151.782120:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699151.782125:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.782127:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.782130:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699151.782133:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699151.782140:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699151.782147:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699151.782150:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699151.782153:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699151.782158:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699151.782161:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699151.782163:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699151.782180:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787699151.782188:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787699151.782215:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00000040:2.0:1787699151.782218:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699151.782230:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:2.0:1787699151.782235:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7062) 00000100:00000040:0.0:1787699151.782240:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126427776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699151.782259:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126427776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699151.782271:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699151.782306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782317:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699151.782331:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782339:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699151.782351:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699151.782357:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782369:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699151.782374:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699151/real 1787699151] req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782386:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782411:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699151.782446:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699151.782451:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699151.782455:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782463:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126427904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699151.782468:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699151.782488:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782496:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699151.782507:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699151.782514:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782524:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699151.782529:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699151/real 1787699151] req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782541:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782549:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699151.782558:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699151.782561:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699151.782572:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782580:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699151.782586:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782595:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699151.782609:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782616:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699151.782626:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699151.782640:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782650:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699151.782656:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699151/real 1787699151] req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782668:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.782676:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699151.782684:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699151.782688:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699151.782692:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782699:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699151.782705:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.782714:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699151.783594:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.783602:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699151.783619:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699151.783628:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699176 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.783639:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699151.783644:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699151/real 1787699151] req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.783657:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699151.783667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699151.783675:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699151.783678:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699151.783682:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699151.783691:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699151.783698:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126427904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699151.783705:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699151.783712:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126428032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699151.783720:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699151.783738:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126428160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699151.783746:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699151.783750:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126428288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699176 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699151.783757:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699152.293548:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699152.293548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699152.293670:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699152.424090:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699153.317534:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699153.317572:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699153.317985:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699153.445678:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699153.701575:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301727087, at 4301727104 00000100:00000040:2.0:1787699153.704029:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301727087, at 4301727106 00000100:00100000:1.0:1787699154.341558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699154.341558:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699154.341668:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699154.471771:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699155.365553:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699155.365553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699155.365825:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699155.494399:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699156.389565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699156.389597:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699156.389624:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699156.517631:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699156.901628:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699156.901675:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699156.901680:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699156.901693:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7057 00000100:00080000:3.0:1787699156.901707:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699156.901711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.901716:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.901721:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699156.901726:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699156.901730:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.901733:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.901737:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699156.901740:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699156.901760:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699156.901793:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699156.901812:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699156.901841:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699156.901845:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699156.901847:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699156.901856:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7057 00000100:00080000:3.0:1787699156.901861:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699156.901863:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.901867:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.901869:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699156.901872:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699156.901875:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.901878:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.901881:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699156.901883:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699156.901895:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699156.901901:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699156.901911:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699156.901921:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699156.901929:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699156.901933:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699156.901935:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699156.901939:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7057 00000100:00080000:3.0:1787699156.901943:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699156.901946:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.901949:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.901951:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699156.901960:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699156.901963:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.901965:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.901968:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699156.901970:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699156.901976:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699156.901980:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699156.901993:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699156.902002:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699156.902010:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699156.902014:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699156.902015:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699156.902019:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7057 00000100:00080000:3.0:1787699156.902023:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699156.902025:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.902028:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.902030:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699156.902033:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699156.902035:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.902037:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.902039:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699156.902042:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699156.902046:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699156.902050:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699156.902057:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699156.902066:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699156.902073:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699156.902076:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699156.902078:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699156.902081:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7057 00000100:00080000:3.0:1787699156.902084:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699156.902086:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.902088:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.902090:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699156.902092:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699156.902094:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699156.902097:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699156.902099:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699156.902101:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699156.902104:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699156.902108:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699156.902114:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699156.902123:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699156.902133:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7068) 00000100:00000040:0.0:1787699156.902227:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902243:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126428416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699156.902277:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902285:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699156.902331:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699156.902353:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699156/real 1787699156] req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902413:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902466:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699156.902474:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699156.902478:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699156.902482:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902490:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126428416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699156.902497:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902505:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699156.902520:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902528:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699156.902540:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699156.902548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902560:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699156.902566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699156/real 1787699156] req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902578:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902587:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699156.902594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699156.902597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699156.902601:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902608:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699156.902613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902620:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699156.902632:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902650:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699156.902662:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699156.902669:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902678:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699156.902683:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699156/real 1787699156] req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902699:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902707:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699156.902715:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699156.902718:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699156.902721:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902728:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699156.902733:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902741:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699156.902755:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902762:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699156.902774:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699156.902780:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902789:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699156.902795:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699156/real 1787699156] req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902811:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902818:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699156.902825:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699156.902827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699156.902830:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902836:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699156.902841:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902848:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699156.902858:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902864:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699156.902873:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699156.902879:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699181 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902887:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699156.902891:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699156/real 1787699156] req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902900:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699156.902907:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699156.902914:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699156.902916:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699156.902920:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699156.902927:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126428928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699156.902932:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126428416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699156.902942:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699156.902951:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126428544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699156.902958:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699156.902963:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126428672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699156.902970:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699156.902975:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126428800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699156.902981:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699156.902986:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126428928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699181 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699156.902991:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699157.413579:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699157.413579:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699157.413691:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699157.541671:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699158.437538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699158.437608:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699158.437923:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699158.567000:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699158.821574:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301732207, at 4301732224 00000100:00000040:3.0:1787699158.821671:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301732207, at 4301732224 00000100:00100000:2.0:1787699159.461592:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699159.461593:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699159.461653:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699159.589664:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699160.485548:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699160.485569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699160.485569:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699160.613874:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699161.509596:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699161.509763:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699161.510004:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699161.639764:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699162.024758:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699162.024770:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699162.024775:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699162.024787:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7063 00000100:00080000:3.0:1787699162.024800:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699162.024803:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.024809:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.024813:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699162.024819:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699162.024822:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.024825:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.024828:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699162.024831:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699162.024850:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699162.024884:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699162.024902:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699162.024931:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699162.024935:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699162.024937:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699162.024942:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7063 00000100:00080000:3.0:1787699162.024946:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699162.024948:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.024951:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.024953:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699162.024956:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699162.024959:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.024961:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.024963:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699162.024966:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699162.024976:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699162.024982:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699162.024992:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699162.025001:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699162.025009:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699162.025012:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699162.025013:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699162.025017:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7063 00000100:00080000:3.0:1787699162.025021:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699162.025023:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.025026:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.025027:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699162.025030:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699162.025032:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.025035:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.025037:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699162.025039:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699162.025043:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699162.025047:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699162.025054:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699162.025064:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699162.025072:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699162.025075:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699162.025077:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699162.025080:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7063 00000100:00080000:3.0:1787699162.025084:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699162.025087:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.025090:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.025092:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699162.025094:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699162.025096:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.025099:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.025101:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699162.025104:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699162.025109:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699162.025113:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699162.025121:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699162.025131:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699162.025139:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699162.025143:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699162.025145:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699162.025148:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7063 00000100:00080000:3.0:1787699162.025151:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699162.025154:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.025156:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.025159:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699162.025161:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699162.025164:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699162.025167:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699162.025169:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699162.025171:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699162.025176:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699162.025180:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699162.025188:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699162.025197:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699162.025206:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7073) 00000100:00000040:0.0:1787699162.025297:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025313:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126429056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699162.025346:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025355:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699162.025403:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699162.025465:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025505:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025521:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025529:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699162.025537:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699162.025541:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699162.025544:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025562:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126429056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699162.025569:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025576:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699162.025591:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025598:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699162.025610:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699162.025617:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025627:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699162.025633:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025645:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025653:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699162.025660:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699162.025664:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699162.025667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025675:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699162.025681:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025690:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699162.025702:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025709:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699162.025721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699162.025728:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025737:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699162.025743:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025754:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699162.025771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699162.025774:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699162.025778:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699162.025791:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025799:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699162.025811:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025818:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699162.025830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699162.025836:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025852:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699162.025858:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025870:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025879:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699162.025886:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699162.025889:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699162.025893:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025900:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699162.025905:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025914:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699162.025938:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025945:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699162.025957:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699162.025964:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699187 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025973:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699162.025979:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699162/real 1787699162] req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.025990:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699162.025998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699162.026005:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699162.026008:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699162.026012:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699162.026020:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699162.026027:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126429056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699162.026038:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699162.026047:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126429184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699162.026055:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699162.026060:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126429312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699162.026067:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699162.026072:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126429440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699162.026079:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699162.026085:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126429568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699187 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699162.026092:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699162.533628:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699162.533910:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699162.535068:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699162.661623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699163.557557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699163.558108:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699163.558334:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699163.685851:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699163.941574:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301737331, at 4301737344 00000100:00000040:3.0:1787699163.941576:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301737331, at 4301737344 00000100:00100000:2.0:1787699164.581559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699164.581576:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699164.581688:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699164.709654:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699165.605547:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699165.605547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699165.605747:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699165.733640:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699166.629528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699166.629529:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699166.629680:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699166.757664:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699167.141628:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699167.141641:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699167.141647:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699167.141660:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7068 00000100:00080000:1.0:1787699167.141672:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699167.141676:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.141681:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.141686:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699167.141692:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699167.141695:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.141698:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.141701:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699167.141704:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699167.141721:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699167.141753:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699167.141770:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699167.141794:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699167.141798:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699167.141799:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699167.141804:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7068 00000100:00080000:1.0:1787699167.141807:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699167.141811:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.141813:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.141815:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699167.141819:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699167.141822:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.141826:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.141828:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699167.141832:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699167.141842:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699167.141850:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699167.141854:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699167.141862:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699167.141878:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126429696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787699167.141879:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699167.141893:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699167.141897:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699167.141899:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699167.141903:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7068 00000100:00080000:1.0:1787699167.141907:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699167.141910:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.141913:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.141915:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699167.141919:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699167.141920:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699167.141924:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.141931:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.141936:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00000040:0.0:1787699167.141941:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699167.141942:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699167.141950:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699167.141955:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699167.141965:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699167.141975:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699167.141984:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699167.141989:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699167.141992:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787699167.141993:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699167.141999:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7068 00000100:00080000:1.0:1787699167.142003:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699167.142006:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.142009:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.142011:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699167.142013:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699167.142016:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.142019:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.142021:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699167.142024:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:0.0:1787699167.142029:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699167.142030:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699167.142040:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699167.142051:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699167.142060:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699167.142069:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699167.142073:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699167.142074:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787699167.142079:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699167.142081:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7068 00000100:00080000:1.0:1787699167.142088:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699167.142094:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.142099:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.142102:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000100:00100000:0.0:1787699167.142104:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699167.142113:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699167.142118:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699167.142119:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699167.142128:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699167.142132:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699167.142133:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787699167.142136:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:0.0:1787699167.142141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000008:00100000:1.0:1787699167.142143:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:0.0:1787699167.142149:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699167.142149:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699167.142163:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699167.142164:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126429696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699167.142175:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126429696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699167.142182:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699167.142198:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787699167.142200:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7078) 00000100:00000040:0.0:1787699167.142220:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142231:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699167.142243:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142248:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699167.142257:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699167.142262:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142269:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699167.142273:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142280:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699167.142290:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699167.142292:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699167.142295:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142300:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699167.142303:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699167.142327:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142335:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699167.142346:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699167.142352:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142360:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699167.142364:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142371:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142379:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699167.142387:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699167.142391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699167.142394:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126429952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699167.142409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142448:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699167.142460:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142478:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699167.142487:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699167.142492:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142498:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699167.142502:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142514:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142523:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699167.142531:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699167.142534:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699167.142537:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142545:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699167.142551:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142560:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699167.142569:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142574:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699167.142581:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699167.142585:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699192 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142591:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699167.142594:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699167/real 1787699167] req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142601:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699167.142606:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699167.142610:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699167.142612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699167.142614:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699167.142618:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699167.142620:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126429824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699167.142629:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699167.142633:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126429952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699167.142638:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699167.142648:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126430080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699167.142655:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699167.142661:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126430208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699192 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699167.142667:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699167.653614:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699167.653627:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699167.653839:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699167.782191:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699168.677590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699168.677590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699168.677591:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699168.805709:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699169.061593:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301742447, at 4301742464 00000100:00000040:2.0:1787699169.061630:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301742447, at 4301742464 00000100:00100000:3.0:1787699169.701568:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699169.701744:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699169.704937:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699169.829826:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699170.725538:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699170.725642:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699170.725651:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699170.854063:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699171.749559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699171.749612:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699171.750235:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699171.877688:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699172.262465:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699172.262477:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699172.262483:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699172.262498:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7073 00000100:00080000:3.0:1787699172.262526:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699172.262530:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.262536:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.262541:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699172.262547:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699172.262551:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.262554:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.262565:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699172.262569:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699172.262589:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699172.262632:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699172.262657:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699172.262685:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699172.262689:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699172.262691:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699172.262696:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7073 00000100:00080000:3.0:1787699172.262700:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699172.262704:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.262708:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699172.262783:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.262805:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126430336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699172.262842:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.262851:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699172.262892:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699172.262910:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.262946:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699172/real 1787699172] req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.262962:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.262972:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699172.262981:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699172.262985:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699172.262989:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.262997:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126430336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699172.263004:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126430336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699172.263014:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000020:00000040:3.0:1787699172.263586:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699172.263593:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699172.263599:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.263602:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.263604:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699172.263607:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699172.263619:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699172.263626:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699172.263639:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699172.263649:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699172.264058:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699172.264062:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699172.264064:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699172.264070:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7073 00000100:00080000:3.0:1787699172.264076:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699172.264078:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.264081:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.264084:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699172.264086:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699172.264089:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.264091:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.264093:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699172.264095:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699172.264100:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699172.264105:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699172.264117:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699172.264130:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699172.264140:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699172.264144:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699172.264145:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699172.264149:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7073 00000100:00080000:3.0:1787699172.264153:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699172.264156:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.264158:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.264160:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699172.264163:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699172.264166:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.264169:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.264171:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699172.264174:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699172.264179:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699172.264184:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699172.264193:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699172.264202:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699172.264210:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699172.264212:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699172.264214:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699172.264217:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7073 00000100:00080000:3.0:1787699172.264222:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699172.264225:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.264228:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.264230:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699172.264232:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699172.264235:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699172.264238:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699172.264241:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699172.264244:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699172.264249:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699172.264253:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699172.264260:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699172.264270:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699172.264280:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7083) 00000100:00000040:0.0:1787699172.265236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.265254:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699172.265281:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.265290:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699172.265318:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699172.265334:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.265348:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699172.265356:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699172/real 1787699172] req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.265390:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.265401:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699172.265410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699172.265414:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699172.265436:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.265445:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699172.265452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.265460:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699172.265473:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.265481:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699172.265548:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699172.265558:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.265568:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699172.265574:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699172/real 1787699172] req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.265594:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.267434:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699172.267448:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699172.267452:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699172.267458:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.267466:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699172.267474:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.267485:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699172.267509:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.267518:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699172.267541:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699172.267552:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.267564:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699172.267571:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699172/real 1787699172] req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.267591:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.267599:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699172.267606:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699172.267610:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699172.267613:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.267621:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699172.267627:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.267635:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699172.267648:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.267655:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699172.267668:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699172.267675:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699197 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.267684:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699172.267690:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699172/real 1787699172] req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.267701:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699172.267709:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699172.267716:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699172.267719:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699172.267723:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699172.267730:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126430848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699172.267736:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126430464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699172.267747:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699172.267756:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126430592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699172.267763:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699172.267768:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126430720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699172.267775:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699172.267781:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126430848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699197 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699172.267786:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699172.773753:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699172.774098:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699172.774205:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699172.902047:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699173.797739:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699173.797839:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699173.798053:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699173.926059:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699174.181590:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301747568, at 4301747584 00000100:00000040:2.0:1787699174.181591:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301747568, at 4301747584 00000100:00100000:1.0:1787699174.821547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699174.821547:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699174.821560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699174.949872:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699175.845596:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699175.846394:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699175.846394:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699175.973636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699176.869549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699176.869549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699176.869585:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699176.998530:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699177.381609:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699177.381623:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699177.381629:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699177.381643:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7078 00000100:00080000:0.0:1787699177.381657:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699177.381661:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.381666:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.381670:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699177.381676:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699177.381680:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.381683:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.381686:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699177.381690:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699177.381709:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699177.381756:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699177.381773:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699177.381804:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699177.381809:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699177.381810:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699177.381815:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7078 00000100:00080000:0.0:1787699177.381819:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699177.381822:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.381824:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.381828:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699177.381831:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699177.381834:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.381836:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.381838:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699177.381840:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699177.381851:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699177.381856:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699177.381866:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699177.381875:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699177.381883:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699177.381885:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699177.381888:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699177.381891:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7078 00000100:00080000:0.0:1787699177.381895:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699177.381897:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.381900:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.381903:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699177.381906:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699177.381909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.381912:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.381914:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699177.381917:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699177.381922:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00000040:3.0:1787699177.381924:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699177.381930:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699177.381941:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126430976:192.168.201.136@tcp:250:kworker.0 00000100:00100000:0.0:1787699177.381945:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699177.381991:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.381998:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699177.382042:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699177.382062:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382100:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699177/real 1787699177] req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382114:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382121:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699177.382127:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699177.382131:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699177.382135:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382150:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126430976:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699177.382157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699177.382193:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382201:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699177.382213:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699177.382220:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382238:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699177.382243:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699177/real 1787699177] req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382251:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699177.382261:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699177.382263:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699177.382265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431104:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699177.382278:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126430976/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699177.382289:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699177.382314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126431104/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699177.382321:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699177.382568:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000fd30bbc5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699177.382588:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699177.382592:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699177.382594:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699177.382598:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7078 00000100:00080000:0.0:1787699177.382603:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699177.382606:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.382609:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.382612:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699177.382615:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699177.382619:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.382622:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.382625:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699177.382628:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:3.0:1787699177.382632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787699177.382636:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699177.382646:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699177.382651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431232:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787699177.382665:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699177.382671:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699177.382688:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000aa4925a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382689:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699177.382701:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699177.382705:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787699177.382709:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699177.382721:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382733:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699177.382741:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699177/real 1787699177] req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382753:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382763:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699177.382766:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699177.382775:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7078 00000100:00080000:3.0:1787699177.382777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699177.382781:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787699177.382785:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699177.382789:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699177.382795:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.382797:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699177.382799:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699177.382802:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699177.382803:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699177.382806:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699177.382813:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699177.382819:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:3.0:1787699177.382822:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431232:192.168.201.136@tcp:38:kworker.0 00000008:00100000:0.0:1787699177.382827:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:3.0:1787699177.382832:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95500 x1874531126431232/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699177.382835:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000020:00000040:3.0:1787699177.382847:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699177.382851:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699177.382861:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699177.382862:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ab71e1f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699177.382890:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7088) 00000100:00100000:3.0:1787699177.382892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699177.382912:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382920:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699177.382933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699177.382940:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382950:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699177.382956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699177/real 1787699177] req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.382967:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.382976:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699177.382990:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699177.382994:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699177.382998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.383006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431360:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699177.383012:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94a80 x1874531126431360/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699177.383020:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699177.383029:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.383038:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699177.383050:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.383058:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699177.383068:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699177.383075:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699202 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.383084:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699177.383089:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699177/real 1787699177] req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.383100:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699177.383115:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699177.383123:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699177.383126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699177.383130:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699177.383138:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431488:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699177.383144:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95f80 x1874531126431488/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699202 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699177.383151:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699177.893559:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699177.893559:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699177.894242:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699178.021712:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699178.917560:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699178.917561:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699178.917561:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699179.045646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699179.301567:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301752687, at 4301752704 00000100:00000040:3.0:1787699179.301567:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301752687, at 4301752704 00000100:00100000:0.0:1787699179.941541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699179.941541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699179.941649:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699180.069607:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699180.965518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699180.965672:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699180.966094:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699181.093874:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699181.989505:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699181.989505:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699181.989536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699182.117737:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699182.501581:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699182.501590:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699182.501595:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699182.501603:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7083 00000100:00080000:1.0:1787699182.501613:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699182.501615:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501618:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501621:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699182.501626:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699182.501628:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501630:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501632:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699182.501634:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501646:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699182.501665:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699182.501676:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699182.501694:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699182.501698:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699182.501700:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699182.501704:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7083 00000100:00080000:1.0:1787699182.501708:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699182.501711:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501714:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501717:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699182.501719:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699182.501722:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501725:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501728:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699182.501731:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699182.501746:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699182.501752:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699182.501758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699182.501772:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699182.501776:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126431616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787699182.501794:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699182.501804:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:3.0:1787699182.501806:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699182.501810:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699182.501815:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:3.0:1787699182.501821:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699182.501824:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7083 00000100:00080000:1.0:1787699182.501830:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699182.501833:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501836:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501839:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699182.501841:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699182.501844:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501847:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501850:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000800:00000100:3.0:1787699182.501852:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699182.501856:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699182.501861:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699182.501865:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699182.501868:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699182.501878:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699182.501885:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699182.501891:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699182.501894:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787699182.501895:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699182/real 1787699182] req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699182.501897:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699182.501905:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7083 00000100:00080000:1.0:1787699182.501911:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699182.501913:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501915:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501917:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699182.501919:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00100000:3.0:1787699182.501921:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699182.501923:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.501928:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.501932:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699182.501938:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:3.0:1787699182.501945:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699182.501946:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699182.501954:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:3.0:1787699182.501961:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699182.501965:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00100000:1.0:1787699182.501969:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699182.501970:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.501998:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126431616:192.168.201.136@tcp:250:kworker.0 00000100:00000040:1.0:1787699182.502002:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502007:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126431616/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699182.502013:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699182.502017:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699182.502019:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000020:00000040:3.0:1787699182.502020:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00080000:1.0:1787699182.502023:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7083 00000100:00080000:1.0:1787699182.502027:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699182.502029:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.502031:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.502033:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699182.502034:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699182.502036:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699182.502038:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699182.502039:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699182.502041:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699182.502045:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:3.0:1787699182.502046:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699182.502048:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699182.502061:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787699182.502061:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:1.0:1787699182.502077:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502084:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699182.502089:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7093) 00000100:00000040:3.0:1787699182.502097:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699182.502112:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699182.502121:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502130:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699182.502135:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699182/real 1787699182] req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502145:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502153:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699182.502159:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699182.502162:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699182.502166:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502172:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431744:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699182.502178:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502185:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699182.502198:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502216:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699182.502228:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699182.502235:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502244:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699182.502250:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699182/real 1787699182] req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502264:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699182.502274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699182.502276:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699182.502278:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502282:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126431872:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699182.502285:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502290:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699182.502299:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502303:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699182.502310:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699182.502314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502320:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699182.502323:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699182/real 1787699182] req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502333:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699182.502342:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699182.502344:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699182.502346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502350:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432000:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699182.502353:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126431744/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699182.502357:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699182.502360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126431872/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699182.502364:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699182.502367:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126432000/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699182.502371:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699182.502375:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502380:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699182.502388:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502392:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699182.502397:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699182.502400:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699207 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502405:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699182.502408:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699182/real 1787699182] req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502414:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699182.502436:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699182.502440:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699182.502442:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699182.502444:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699182.502448:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432128:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699182.502451:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126432128/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699207 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699182.502456:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699183.013533:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699183.013533:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699183.013533:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699183.141685:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699184.038293:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699184.038546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699184.038848:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699184.165767:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699184.421530:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301757807, at 4301757824 00000100:00000040:3.0:1787699184.421530:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301757807, at 4301757824 00000100:00100000:0.0:1787699185.061538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699185.061538:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699185.061570:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699185.189624:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699186.085536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699186.085556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699186.085557:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699186.213637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699187.109539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699187.109539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699187.109542:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699187.237621:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699187.621582:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699187.621596:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699187.621601:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699187.621615:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7088 00000100:00080000:2.0:1787699187.621629:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699187.621633:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621638:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621642:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699187.621648:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699187.621652:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621654:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621658:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699187.621661:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621679:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699187.621709:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699187.621722:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699187.621742:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699187.621746:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699187.621748:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699187.621753:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7088 00000100:00080000:2.0:1787699187.621757:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699187.621760:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621762:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621765:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699187.621768:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699187.621772:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621775:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621777:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699187.621781:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00000040:3.0:1787699187.621790:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787699187.621792:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699187.621801:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699187.621816:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126432256:192.168.201.136@tcp:250:kworker.0 00000100:00100000:2.0:1787699187.621823:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699187.621835:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699187.621845:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699187.621849:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699187.621851:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699187.621858:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7088 00000100:00000040:3.0:1787699187.621859:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699187.621863:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699187.621869:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621873:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699187.621874:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621879:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699187.621881:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699187.621884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621887:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699187.621889:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699187.621894:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699187.621898:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699187.621904:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699187.621911:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699187.621918:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699187.621922:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699187.621923:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699187.621927:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7088 00000100:00080000:2.0:1787699187.621930:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699187.621932:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621933:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621935:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699187.621937:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699187.621939:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.621940:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.621942:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699187.621945:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699187.621949:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699187.621954:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699187.621965:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699187.621966:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699187.621979:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699187.621991:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787699187.621992:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432384:192.168.201.136@tcp:38:kworker.0 00000100:00080000:2.0:1787699187.621995:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699187.621997:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699187.622002:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7088 00000100:00080000:2.0:1787699187.622004:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699187.622006:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.622007:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.622009:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699187.622010:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699187.622014:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699187.622016:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699187.622018:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699187.622019:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699187.622023:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00000040:3.0:1787699187.622023:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:2.0:1787699187.622030:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699187.622039:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699187.622040:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:2.0:1787699187.622049:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699187.622057:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7098) 00000100:00000040:3.0:1787699187.622059:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00000040:3.0:1787699187.622068:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699187.622076:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699187.622089:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699187.622096:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699187.622105:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:3.0:1787699187.622113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699187.622133:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432640:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699187.622153:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699187.622160:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699187.622171:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:3.0:1787699187.622178:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699187.622186:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699187.622198:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699187.622203:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699187.622211:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699188.133552:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699188.133553:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699188.133554:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699188.261662:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699189.157525:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699189.157548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699189.157646:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699189.288351:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699189.541570:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301762927, at 4301762944 00000100:00000040:2.0:1787699189.541637:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301762927, at 4301762944 00000100:00100000:0.0:1787699190.181570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699190.181595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699190.183161:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699190.310986:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787699190.694561:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:2.0:1787699190.694617:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787699190.694636:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787699190.694677:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787699190.694697:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787699190.694710:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.694713:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699187/real 1787699190] req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787699190.694732:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:0.0:1787699190.694751:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699190.694764:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699190.694777:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699190.694784:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699190.694789:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.694799:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126432256:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699190.694810:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126432256/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699190.694824:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699190.694837:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699187/real 1787699190] req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699190.696011:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787699190.696031:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787699190.696043:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787699190.696054:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:2.0:1787699190.696062:0:2386:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699212 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698172:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699190.698189:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699190.698202:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699190.698207:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699190.698211:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698220:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432384:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787699190.698229:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699187/real 1787699190] req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698242:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699190.698251:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699190.698259:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699190.698262:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699190.698267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698275:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432512:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699190.698283:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126432384/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699190.698297:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699190.698306:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126432512/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699190.698319:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699190.698327:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699187/real 1787699190] req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699190.698348:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699190.698355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699190.698358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699190.698362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698370:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432640:192.168.201.136@tcp:8:kworker.0 00000100:00100000:0.0:1787699190.698376:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699187/real 1787699190] req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698388:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699190.698397:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699190.698404:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699190.698407:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699190.698412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699190.698437:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126432768:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699190.698444:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126432640/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699190.698452:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699190.698458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126432768/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699212 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699190.698464:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699191.205522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699191.206120:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699191.206726:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699191.333653:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787699192.229518:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699192.229736:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699192.230555:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:2.0:1787699192.358814:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699192.741555:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699192.741570:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699192.741576:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699192.741589:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7093 00000100:00080000:2.0:1787699192.741603:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699192.741607:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741611:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741615:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699192.741620:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699192.741623:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741625:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741628:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699192.741631:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741645:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699192.741677:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699192.741693:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699192.741719:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699192.741723:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699192.741725:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699192.741730:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7093 00000100:00080000:2.0:1787699192.741734:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699192.741737:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741740:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741743:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699192.741746:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699192.741749:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741752:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741754:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699192.741756:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699192.741767:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699192.741772:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699192.741783:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699192.741793:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699192.741801:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699192.741804:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699192.741806:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699192.741811:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7093 00000100:00080000:2.0:1787699192.741814:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699192.741817:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741820:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741822:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699192.741824:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699192.741827:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741830:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741832:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699192.741835:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699192.741839:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699192.741843:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699192.741851:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699192.741860:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699192.741869:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699192.741872:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699192.741874:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699192.741878:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7093 00000100:00080000:2.0:1787699192.741881:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699192.741884:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741886:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741888:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699192.741891:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699192.741894:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741897:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741899:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699192.741902:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699192.741907:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699192.741912:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699192.741921:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699192.741930:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699192.741938:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699192.741941:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699192.741943:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699192.741947:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7093 00000100:00080000:2.0:1787699192.741951:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699192.741953:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741956:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741958:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699192.741961:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699192.741964:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699192.741966:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699192.741968:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699192.741971:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699192.741975:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699192.741979:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699192.741988:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699192.741998:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699192.742008:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7103) 00000100:00000040:0.0:1787699192.742452:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742472:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126432896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699192.742511:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742519:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699192.742568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699192.742589:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699192/real 1787699192] req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742651:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742661:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699192.742677:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699192.742682:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699192.742687:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742696:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126432896:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699192.742703:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742712:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699192.742727:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742735:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699192.742747:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699192.742754:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742766:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699192.742772:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699192/real 1787699192] req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742783:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742792:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699192.742799:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699192.742802:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699192.742806:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742815:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433024:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699192.742822:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742832:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699192.742845:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742854:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699192.742863:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699192.742869:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742891:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699192.742898:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699192/real 1787699192] req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.742910:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.742918:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699192.744023:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699192.744042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699192.744051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.744063:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433152:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699192.744072:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.744082:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699192.744299:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.744309:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699192.744339:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699192.744352:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.744364:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699192.744371:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699192/real 1787699192] req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.744384:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.744393:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699192.744401:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699192.744405:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699192.744409:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.744415:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433280:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699192.744440:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.744450:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699192.744828:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.744836:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699192.744851:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699192.744858:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699217 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.744867:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699192.744985:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699192/real 1787699192] req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.745000:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699192.745008:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699192.745016:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699192.745020:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699192.745024:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699192.745040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433408:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699192.745048:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126432896/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699192.745066:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699192.745077:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126433024/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699192.745084:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699192.745090:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126433152/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699192.745107:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699192.745607:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126433280/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699192.745617:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699192.745623:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126433408/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699217 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699192.745629:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699193.253590:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699193.253590:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699193.253590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699193.381646:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699194.277545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699194.277552:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699194.277735:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699194.405681:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699194.661554:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301768048, at 4301768064 00000100:00000040:2.0:1787699194.661557:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301768048, at 4301768064 00000100:00100000:3.0:1787699195.301544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699195.301772:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699195.305722:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699195.429669:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699196.325509:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699196.325547:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699196.325699:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699196.453688:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699197.349539:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699197.349539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699197.349624:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699197.477812:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699197.862045:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699197.862055:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699197.862059:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699197.862069:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7098 00000100:00080000:0.0:1787699197.862233:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699197.862237:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.862241:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.862244:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699197.862249:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699197.862252:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.862254:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.862256:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699197.862259:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699197.862273:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699197.862889:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699197.862907:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000bc1d571] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699197.862941:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699197.862953:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699197.862955:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699197.862960:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7098 00000100:00080000:0.0:1787699197.862964:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699197.862967:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.862970:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.862973:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699197.862976:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699197.862979:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.862982:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.862985:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699197.862986:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699197.862996:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699197.863000:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699197.863009:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699197.863020:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a94b940] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699197.863025:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699197.863027:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699197.863028:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699197.863030:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7098 00000100:00080000:0.0:1787699197.863033:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699197.863035:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.863036:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.863037:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699197.863039:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699197.863041:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.863043:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.863044:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699197.863046:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699197.863048:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699197.863050:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699197.863055:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699197.863060:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000eff8ba53] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699197.863064:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699197.863066:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699197.863068:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699197.863070:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7098 00000100:00080000:0.0:1787699197.863072:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699197.863086:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.863088:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.863089:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699197.863091:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699197.863093:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.863094:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.863095:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699197.863097:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699197.863100:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699197.863102:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699197.863113:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699197.863118:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000733fdda4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699197.863123:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699197.863125:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699197.863126:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699197.863128:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7098 00000100:00080000:0.0:1787699197.863130:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699197.863132:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.863133:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.863134:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699197.863136:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699197.863138:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699197.863141:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699197.863144:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699197.863146:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699197.863150:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699197.863155:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699197.863162:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699197.863182:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000496e51d4] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699197.863192:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7108) 00000100:00000040:3.0:1787699197.863816:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.863836:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126433536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699197.863870:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.863878:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699197.863927:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699197.863960:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.863996:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699197/real 1787699197] req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864012:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864021:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699197.864029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699197.864033:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699197.864037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864045:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126433536:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699197.864051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864059:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699197.864086:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864093:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699197.864116:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699197.864124:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864134:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699197.864140:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699197/real 1787699197] req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864161:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699197.864488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699197.864493:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699197.864498:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864506:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433664:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699197.864512:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864521:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699197.864536:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864544:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699197.864577:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699197.864582:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864589:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699197.864593:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699197/real 1787699197] req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864600:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864606:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699197.864611:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699197.864612:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699197.864614:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864619:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433792:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699197.864622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864627:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699197.864636:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864640:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699197.864646:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699197.864650:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864660:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699197.864665:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699197/real 1787699197] req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864675:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864682:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699197.864689:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699197.864692:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699197.864696:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864704:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126433920:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699197.864709:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864716:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699197.864725:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864730:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699197.864737:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699197.864741:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699222 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864746:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699197.864750:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699197/real 1787699197] req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864757:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699197.864762:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699197.864766:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699197.864768:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699197.864770:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699197.864774:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434048:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699197.864778:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95880 x1874531126433536/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699197.864785:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699197.864792:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94700 x1874531126433664/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699197.864796:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699197.864800:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95c00 x1874531126433792/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699197.864804:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699197.864806:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96a00 x1874531126433920/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699197.864810:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699197.864814:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96680 x1874531126434048/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699222 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699197.864818:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699198.373575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699198.373575:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699198.373729:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699198.501807:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699199.397527:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699199.397528:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699199.397863:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699199.526515:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699199.781583:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301773169, at 4301773184 00000100:00000040:2.0:1787699199.783268:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301773169, at 4301773185 00000100:00100000:3.0:1787699200.421555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699200.421556:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699200.425990:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699200.549957:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699201.446099:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699201.446099:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699201.446217:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699201.593636:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699202.469572:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699202.469572:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699202.469961:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699202.597754:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699202.983557:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699202.983570:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699202.983575:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699202.983588:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7103 00000100:00080000:0.0:1787699202.983601:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699202.983604:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983610:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983614:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699202.983621:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699202.983625:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983628:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983631:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699202.983635:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983653:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699202.983685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699202.983703:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699202.983730:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699202.983734:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699202.983736:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699202.983741:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7103 00000100:00080000:0.0:1787699202.983745:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699202.983749:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983751:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983754:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699202.983757:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699202.983760:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983764:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983766:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699202.983770:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699202.983784:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699202.983793:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699202.983808:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699202.983819:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699202.983829:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699202.983833:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699202.983834:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699202.983839:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7103 00000100:00080000:0.0:1787699202.983842:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699202.983845:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983848:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983850:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699202.983852:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699202.983855:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983867:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983869:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699202.983871:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699202.983877:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699202.983881:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699202.983889:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699202.983899:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699202.983907:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699202.983910:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699202.983912:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699202.983915:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7103 00000100:00000040:3.0:1787699202.983915:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699202.983919:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699202.983922:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983924:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983926:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699202.983928:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699202.983931:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:3.0:1787699202.983931:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126434176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699202.983934:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983936:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699202.983938:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699202.983942:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699202.983946:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699202.983953:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699202.983961:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.983966:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699202.983969:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699202.983973:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699202.983974:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:3.0:1787699202.983976:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699202.983978:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7103 00000100:00080000:0.0:1787699202.983981:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699202.983983:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983988:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699202.983989:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699202.983992:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699202.983994:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699202.983996:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699202.983998:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699202.984002:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699202.984005:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699202.984012:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699202.984020:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787699202.984024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699202.984031:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7114) 00000100:00000040:3.0:1787699202.984045:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984085:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699202/real 1787699202] req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984103:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.984112:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699202.984121:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699202.984126:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699202.984131:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984139:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126434176:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699202.984146:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984154:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699202.984168:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.984175:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699202.984187:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699202.984194:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.984205:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699202.984211:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699202/real 1787699202] req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984220:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.984228:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699202.984236:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699202.984238:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699202.984242:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984251:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434304:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699202.984256:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984265:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699202.984277:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.984285:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699202.984295:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699202.984302:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.984310:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699202.984316:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699202/real 1787699202] req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984327:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.984335:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699202.984343:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699202.984346:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699202.984349:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.984356:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434432:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699202.984363:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126434176/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699202.984373:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699202.984383:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126434304/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699202.984391:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699202.984396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126434432/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699202.984403:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699202.984415:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.986793:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699202.986828:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.986837:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699202.986884:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699202.986912:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.986928:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699202.986937:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699202/real 1787699202] req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.986953:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.986962:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699202.986973:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699202.986977:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699202.986981:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.986989:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434560:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699202.986997:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.987006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699202.987020:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.987027:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699202.987040:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699202.987047:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699227 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.987057:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699202.987062:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699202/real 1787699202] req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.987073:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699202.987082:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699202.987089:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699202.987093:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699202.987097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699202.987105:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434688:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699202.987111:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126434560/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699202.987123:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699202.987133:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126434688/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699227 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699202.987140:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699203.493541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699203.493742:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699203.493743:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699203.621657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699204.517551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699204.517659:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699204.517735:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699204.646987:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699204.901558:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301778289, at 4301778304 00000100:00000040:3.0:1787699204.904052:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301778289, at 4301778306 00000100:00100000:0.0:1787699205.541592:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699205.541592:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699205.542958:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699205.670072:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699206.565536:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699206.565709:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699206.566058:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699206.693659:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699207.589565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699207.589948:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699207.590023:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699207.717906:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699208.103163:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699208.103176:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699208.103182:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699208.103201:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7109 00000100:00080000:2.0:1787699208.103214:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699208.103218:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103223:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103227:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699208.103232:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699208.103236:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103238:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103242:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699208.103245:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103264:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699208.103297:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699208.103313:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076d2ff4d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699208.103337:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699208.103341:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699208.103343:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699208.103347:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7109 00000100:00080000:2.0:1787699208.103351:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699208.103354:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103357:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103359:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699208.103361:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699208.103365:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103367:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103370:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699208.103373:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699208.103383:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699208.103388:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699208.103397:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699208.103407:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000884f82f2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699208.103415:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699208.103443:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699208.103445:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699208.103449:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7109 00000100:00080000:2.0:1787699208.103452:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699208.103455:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103458:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103460:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699208.103462:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699208.103465:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103467:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103470:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699208.103473:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699208.103478:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699208.103482:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699208.103492:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699208.103501:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4a15651] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699208.103509:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699208.103512:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699208.103514:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699208.103518:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7109 00000100:00080000:2.0:1787699208.103521:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699208.103524:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103526:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103528:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699208.103530:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699208.103533:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103536:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103538:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699208.103541:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699208.103545:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699208.103549:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699208.103557:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699208.103572:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000017f956a8] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699208.103580:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699208.103583:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699208.103584:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699208.103588:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7109 00000100:00080000:2.0:1787699208.103591:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699208.103594:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103596:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103598:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699208.103600:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699208.103603:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699208.103605:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699208.103607:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699208.103610:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699208.103614:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699208.103618:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699208.103624:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699208.103633:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000729819f6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699208.103643:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7119) 00000100:00000040:3.0:1787699208.103761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.103785:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126434816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699208.103817:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.103836:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699208.103877:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699208.103898:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.103930:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.103954:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.103964:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.103974:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.103978:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699208.103981:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.103991:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35adf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126434816:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699208.103997:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699208.104019:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104027:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699208.104036:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699208.104041:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104052:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699208.104057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104068:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104077:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.104085:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.104087:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699208.104091:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104099:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126434944:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699208.104104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104111:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699208.104120:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104127:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699208.104135:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699208.104140:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104149:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699208.104154:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104164:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104172:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.104182:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.104185:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699208.104199:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104206:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435072:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699208.104212:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104220:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699208.104232:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104239:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699208.104248:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699208.104254:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104262:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699208.104267:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104277:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104286:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.104294:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.104297:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699208.104300:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104308:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35aea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435200:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699208.104312:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104320:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699208.104330:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104336:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699208.104345:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699208.104350:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699233 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104359:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699208.104363:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699208/real 1787699208] req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104374:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699208.104382:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699208.104388:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699208.104391:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699208.104395:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699208.104403:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ace00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435328:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699208.104409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35adf80 x1874531126434816/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699208.104447:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699208.104458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af480 x1874531126434944/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699208.104466:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699208.104470:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac000 x1874531126435072/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699208.104478:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699208.104483:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35aea00 x1874531126435200/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699208.104490:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699208.104494:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ace00 x1874531126435328/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699233 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699208.104501:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699208.613540:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699208.613650:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699208.615895:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699208.745273:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699209.637642:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:2.0:1787699209.637642:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699209.637882:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699209.765660:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699210.021545:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301783409, at 4301783424 00000100:00000040:2.0:1787699210.022408:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301783409, at 4301783424 00000100:00100000:1.0:1787699210.661577:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699210.661591:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699210.661713:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699210.791119:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699211.685577:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699211.685707:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699211.685933:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699211.951416:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699212.709584:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699212.710931:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699212.712720:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00010000:00010000:0.0:1787699212.965868:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699213.221691:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699213.221705:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699213.221711:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699213.221724:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7114 00000100:00080000:0.0:1787699213.221738:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699213.221742:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.221747:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.221752:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699213.221757:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699213.221761:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.221764:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.221767:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699213.221770:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699213.221788:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699213.221820:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699213.221837:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.221910:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.221927:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126435456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699213.221970:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.221979:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699213.222025:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699213.222061:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699213.222065:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699213.222067:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699213.222072:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7114 00000100:00080000:0.0:1787699213.222077:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699213.222080:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.222083:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.222086:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699213.222089:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699213.222092:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.222095:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.222097:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699213.222100:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699213.222111:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699213.222116:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699213.222126:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699213.222136:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222170:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222206:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699213/real 1787699213] req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222224:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222235:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699213.222244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699213.222248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699213.222252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222269:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126435456:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699213.222276:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126435456/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699213.222288:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699213.222306:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222316:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699213.222332:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222340:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699213.222352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699213.222360:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222371:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699213.222377:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699213/real 1787699213] req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222389:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222398:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699213.222406:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699213.222409:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699213.222413:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222444:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435584:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699213.222450:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126435584/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699213.222458:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00080000:0.0:1787699213.222564:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699213.222568:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699213.222570:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699213.222574:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7114 00000100:00080000:0.0:1787699213.222578:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699213.222581:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.222584:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.222586:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699213.222589:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699213.222592:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.222595:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.222598:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699213.222602:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699213.222608:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699213.222613:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699213.222625:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699213.222636:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222680:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699213.222714:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222720:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699213.222733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699213.222753:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699213.222757:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699213.222759:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699213.222764:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7114 00000100:00080000:0.0:1787699213.222768:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699213.222771:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.222775:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.222777:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699213.222779:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699213.222782:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.222785:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.222787:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699213.222790:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699213.222795:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699213.222800:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699213.222811:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699213.222820:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222853:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222864:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699213.222869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699213/real 1787699213] req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222881:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222890:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699213.222898:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699213.222901:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699213.222905:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222913:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435712:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699213.222918:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126435712/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699213.222927:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699213.222946:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.222956:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699213.222969:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.222982:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699213.222995:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699213.223003:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.223012:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699213.223018:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699213/real 1787699213] req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.223028:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.223037:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699213.223044:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699213.223047:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699213.223051:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.223059:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435840:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699213.223064:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126435840/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699213.223072:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:0.0:1787699213.223157:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699213.223162:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699213.223164:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699213.223168:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7114 00000100:00080000:0.0:1787699213.223172:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699213.223175:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.223178:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.223180:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699213.223183:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699213.223186:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699213.223188:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699213.223190:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699213.223192:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699213.223197:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699213.223202:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699213.223212:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699213.223222:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.223265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.223274:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699213.223291:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.223298:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699213.223312:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699213.223334:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7124) 00000100:00000040:3.0:1787699213.223373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699238 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.223384:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699213.223390:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699213/real 1787699213] req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.223402:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699213.223411:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699213.223437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699213.223441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699213.223446:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699213.223454:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126435968:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699213.223460:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126435968/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699238 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699213.223468:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699213.733582:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:2.0:1787699213.733751:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699213.733810:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699213.991122:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699214.757497:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699214.758965:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699214.759223:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699215.014026:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699215.142723:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301788527, at 4301788545 00000100:00000040:3.0:1787699215.142868:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301788527, at 4301788545 00000100:00100000:3.0:1787699215.782538:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699215.809994:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699215.810507:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699216.038340:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699216.805775:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699216.869606:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699216.871934:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699217.061755:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699217.829671:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699217.893566:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699217.893750:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699218.085710:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699218.341638:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699218.341651:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699218.341655:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699218.341668:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7119 00000100:00080000:1.0:1787699218.341681:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699218.341685:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.341690:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.341694:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699218.341700:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699218.341703:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.341706:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.341710:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699218.341713:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699218.341731:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699218.341764:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699218.341782:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.341866:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.341881:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126436096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699218.341915:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.341925:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699218.341974:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699218.342009:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699218.342014:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699218.342016:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699218.342021:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7119 00000100:00080000:1.0:1787699218.342025:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699218.342028:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.342031:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.342034:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699218.342037:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699218.342040:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.342043:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.342046:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699218.342048:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699218.342059:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699218.342064:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699218.342074:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699218.342085:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.344809:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.344874:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699218/real 1787699218] req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.344901:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.344912:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699218.344924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699218.344929:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699218.344935:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.344951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126436096:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699218.344960:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126436096/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699218.344975:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699218.344997:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.345011:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699218.345039:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.345048:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699218.345080:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699218.345092:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.345102:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699218.345108:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699218/real 1787699218] req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.345120:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.345129:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699218.345138:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699218.345141:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699218.345145:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.345737:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436224:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699218.345749:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126436224/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699218.345758:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00080000:1.0:1787699218.346410:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699218.346428:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699218.346432:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699218.346438:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7119 00000100:00080000:1.0:1787699218.346447:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699218.346450:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.346454:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.346459:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699218.346465:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699218.346469:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.346471:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.346654:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787699218.346661:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699218.346673:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699218.346684:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699218.346708:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699218.346764:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.346811:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.346823:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699218.346847:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.346856:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699218.346885:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699218.347010:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699218.347015:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699218.347017:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699218.347022:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7119 00000100:00080000:1.0:1787699218.347026:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699218.347029:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.347033:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.347035:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699218.347038:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699218.347041:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.347044:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.347046:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699218.347049:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699218.347055:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699218.347060:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699218.347070:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699218.347081:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.347538:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.347556:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699218.347566:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699218/real 1787699218] req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.347582:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.347592:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699218.347612:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699218.347617:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699218.347621:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.347630:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436352:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699218.347637:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126436352/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699218.347648:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699218.347664:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.347673:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699218.347692:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.347701:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699218.347721:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699218.347731:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.347740:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699218.347746:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699218/real 1787699218] req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.347758:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.347767:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699218.347782:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699218.347786:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699218.347790:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.347798:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436480:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699218.347803:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126436480/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699218.347811:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:1.0:1787699218.348004:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699218.348009:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699218.348011:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699218.348016:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7119 00000100:00080000:1.0:1787699218.348022:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699218.348025:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.348029:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.348033:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699218.348036:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699218.348039:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699218.348042:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699218.348045:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699218.348048:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699218.348055:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699218.348061:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699218.348076:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699218.348087:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699218.348098:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7129) 00000100:00000040:3.0:1787699218.348387:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.348397:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699218.348652:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.348679:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699218.348797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699218.348958:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699243 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.349014:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699218.349057:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699218/real 1787699218] req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.349135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699218.352792:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699218.352807:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699218.352812:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699218.352817:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699218.352826:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436608:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699218.352836:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126436608/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699243 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699218.352852:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699218.853678:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699218.917641:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699218.918041:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699219.109799:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699219.877721:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699219.941603:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699219.941860:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699220.134472:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699220.261707:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301793650, at 4301793664 00000100:00000040:2.0:1787699220.263077:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301793650, at 4301793665 00000100:00100000:3.0:1787699220.901635:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699220.965586:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699220.966621:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699221.157725:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699221.925645:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699221.989518:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699221.989523:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:0.0:1787699222.181759:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699222.949569:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699223.013623:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699223.013835:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699223.208882:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699223.461690:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699223.461702:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699223.461706:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699223.461716:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7124 00000100:00080000:3.0:1787699223.461727:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699223.461730:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.461734:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.461739:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699223.461744:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699223.461747:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.461750:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.461753:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699223.461756:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699223.461769:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699223.461795:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699223.461810:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699223.461841:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699223.461844:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699223.461846:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699223.461850:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7124 00000100:00080000:3.0:1787699223.461854:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699223.462198:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.462213:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126436736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699223.462240:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.462249:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699223.462284:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699223.462298:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.462330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699223/real 1787699223] req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.462344:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.462352:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699223.462360:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699223.462363:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699223.462367:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.462380:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126436736:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699223.462386:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126436736/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699223.462395:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462476:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462479:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462482:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699223.462484:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699223.462487:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462489:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462491:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699223.462493:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699223.462503:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699223.462518:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699223.462531:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699223.462540:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699223.462556:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699223.462559:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699223.462561:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699223.462564:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7124 00000100:00080000:3.0:1787699223.462569:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699223.462571:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462573:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462575:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699223.462577:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699223.462579:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462581:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462583:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699223.462585:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699223.462589:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699223.462593:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699223.462600:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699223.462608:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699223.462621:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699223.462624:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699223.462626:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699223.462629:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7124 00000100:00080000:3.0:1787699223.462632:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699223.462634:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462637:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462639:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699223.462641:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699223.462643:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462645:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462647:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699223.462650:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699223.462654:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699223.462657:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699223.462665:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699223.462673:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699223.462679:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699223.462682:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699223.462683:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699223.462686:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7124 00000100:00080000:3.0:1787699223.462689:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699223.462692:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462694:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462696:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699223.462698:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699223.462700:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699223.462702:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699223.462704:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699223.462707:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699223.462711:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699223.462715:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699223.462723:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699223.462731:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699223.462740:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7134) 00000100:00000040:0.0:1787699223.463177:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.463188:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699223.463206:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.463213:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699223.463826:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699223.463842:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464563:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699223.464577:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699223/real 1787699223] req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464596:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464606:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699223.464615:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699223.464619:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699223.464624:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464632:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436864:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699223.464640:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464651:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699223.464670:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464678:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699223.464699:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699223.464708:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464718:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699223.464724:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699223/real 1787699223] req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464735:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464743:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699223.464751:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699223.464754:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699223.464758:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464772:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126436992:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699223.464778:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464787:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699223.464800:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464807:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699223.464818:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699223.464824:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464833:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699223.464839:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699223/real 1787699223] req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464869:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699223.464875:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699223.464879:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699223.464882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464890:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437120:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699223.464896:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464905:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699223.464917:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464925:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699223.464936:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699223.464952:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699248 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464961:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699223.464967:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699223/real 1787699223] req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.464977:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699223.464986:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699223.464992:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699223.464995:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699223.464998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699223.465006:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437248:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699223.465011:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126436864/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699223.465027:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699223.465042:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126436992/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699223.465049:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699223.465055:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126437120/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699223.465060:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699223.465065:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126437248/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699248 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699223.465072:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699223.973534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699224.037746:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699224.037853:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699224.229775:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699224.997495:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699225.061593:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699225.061647:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699225.253596:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699225.381502:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301798767, at 4301798784 00000100:00000040:3.0:1787699225.383637:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301798767, at 4301798786 00000100:00100000:3.0:1787699226.022397:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699226.085568:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699226.085813:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699226.277637:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699227.045849:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699227.109564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699227.109564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699227.301655:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699228.069603:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699228.133595:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699228.133721:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699228.326212:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699228.581649:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699228.581665:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699228.581671:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699228.581686:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7129 00000100:00080000:0.0:1787699228.581702:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699228.581706:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.581711:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.581715:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699228.581721:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699228.581725:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.581727:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.581731:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699228.581735:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699228.581756:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699228.581795:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699228.581814:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000005a55842c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.581880:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.581896:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126437376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699228.581935:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.581943:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699228.581990:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:0.0:1787699228.582027:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699228.582031:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699228.582033:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699228.582039:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7129 00000100:00080000:0.0:1787699228.582043:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699228.582046:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582049:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582051:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699228.582055:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699228.582058:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582061:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582063:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699228.582065:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699228.582077:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699228.582083:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699228.582094:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699228.582104:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000006ac7062] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582140:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582182:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699228/real 1787699228] req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582199:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582209:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699228.582218:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699228.582221:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699228.582226:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582234:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126437376:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699228.582241:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96d80 x1874531126437376/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699228.582251:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699228.582269:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699228.582296:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582305:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699228.582317:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699228.582325:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582335:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699228.582341:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699228/real 1787699228] req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582353:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699228.582369:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699228.582372:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699228.582376:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582384:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a96300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437504:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699228.582390:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a96300 x1874531126437504/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699228.582398:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00080000:0.0:1787699228.582499:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699228.582503:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699228.582505:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699228.582511:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7129 00000100:00080000:0.0:1787699228.582516:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699228.582519:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582522:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582525:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699228.582528:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699228.582531:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582533:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582536:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699228.582538:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699228.582544:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699228.582549:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699228.582559:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699228.582570:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000032656b08] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699228.582578:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699228.582582:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699228.582584:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699228.582588:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7129 00000100:00080000:0.0:1787699228.582592:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699228.582594:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582597:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582599:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699228.582602:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699228.582605:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582608:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582610:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699228.582613:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699228.582618:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699228.582622:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699228.582631:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699228.582640:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002bb6829f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699228.582648:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699228.582651:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699228.582653:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699228.582656:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7129 00000100:00080000:0.0:1787699228.582659:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699228.582662:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582664:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582667:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699228.582669:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699228.582672:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699228.582675:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699228.582677:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699228.582680:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699228.582684:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699228.582688:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699228.582696:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699228.582704:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f0e999e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699228.582713:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7139) 00000100:00000040:3.0:1787699228.582770:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582786:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699228.582803:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582810:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699228.582823:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699228.582830:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582840:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699228.582845:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699228/real 1787699228] req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582856:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582865:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699228.582874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699228.582877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699228.582881:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437632:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699228.582895:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582903:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699228.582916:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582923:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699228.582932:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699228.582938:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582947:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699228.582952:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699228/real 1787699228] req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582963:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.582971:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699228.582979:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699228.582982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699228.582985:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.582992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437760:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699228.582998:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.583006:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699228.583018:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.583031:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699228.583043:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699228.583049:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699253 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.583059:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699228.583064:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699228/real 1787699228] req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.583076:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699228.583084:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699228.583091:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699228.583094:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699228.583098:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699228.583107:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126437888:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699228.583113:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94000 x1874531126437632/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699228.583121:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699228.583128:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97480 x1874531126437760/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699228.583136:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699228.583141:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97800 x1874531126437888/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699253 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699228.583149:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699229.093532:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699229.157545:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699229.157732:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699229.350554:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699230.117579:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699230.181541:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699230.181541:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699230.373604:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699230.501829:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301803887, at 4301803904 00000100:00000040:3.0:1787699230.501859:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301803887, at 4301803904 00000100:00100000:2.0:1787699231.142312:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699231.205548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699231.205658:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699231.399586:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699232.165608:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699232.229536:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699232.229697:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699232.422311:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699233.189546:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699233.253944:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699233.254051:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699233.445686:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699233.701710:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699233.701723:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699233.701729:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699233.701743:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7134 00000100:00080000:3.0:1787699233.701758:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699233.701761:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.701766:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.701771:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699233.701777:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699233.701780:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.701783:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.701786:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699233.701789:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699233.701808:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699233.701842:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699233.701859:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.701923:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.701940:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126438016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699233.701974:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.701984:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699233.702031:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699233.702065:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699233.702070:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699233.702072:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699233.702077:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7134 00000100:00080000:3.0:1787699233.702082:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699233.702085:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.702087:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.702090:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699233.702093:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699233.702096:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.702098:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.702101:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699233.702103:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699233.702114:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699233.702118:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699233.702129:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699233.702139:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.702172:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.702225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699233/real 1787699233] req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.702244:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.702253:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699233.702262:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699233.702277:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699233.702282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.702291:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126438016:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699233.702299:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126438016/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699233.702310:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699233.702329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.702370:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699233.702386:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.702394:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699233.702408:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699233.702416:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.702453:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699233.702459:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699233/real 1787699233] req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.702471:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.702480:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699233.702488:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699233.702491:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699233.702495:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.702510:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438144:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699233.702515:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126438144/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699233.702523:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00080000:3.0:1787699233.702777:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699233.702782:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699233.702784:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699233.702789:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7134 00000100:00080000:3.0:1787699233.702794:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699233.702797:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.702799:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.702802:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699233.702805:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699233.702808:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.702811:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.702814:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699233.702816:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699233.702822:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699233.702827:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699233.702838:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699233.702849:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.702891:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.702900:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699233.702914:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.702921:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699233.702935:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699233.702955:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699233.702959:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699233.702961:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699233.702966:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7134 00000100:00080000:3.0:1787699233.702970:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699233.702972:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.702975:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.702977:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699233.702980:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699233.702983:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.702985:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.702988:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699233.702991:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699233.702996:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699233.703001:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699233.703010:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699233.703020:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703050:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703062:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699233.703068:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699233/real 1787699233] req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703081:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703090:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699233.703098:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699233.703101:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699233.703106:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703114:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438272:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699233.703120:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126438272/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699233.703128:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699233.703137:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703146:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699233.703159:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703166:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699233.703193:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699233.703201:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703210:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699233.703216:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699233/real 1787699233] req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703228:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703236:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699233.703244:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699233.703248:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699233.703252:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703260:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438400:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699233.703326:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126438400/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699233.703343:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00080000:3.0:1787699233.703488:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699233.703493:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699233.703495:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699233.703499:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7134 00000100:00080000:3.0:1787699233.703504:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699233.703507:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.703510:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.703513:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699233.703515:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699233.703518:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699233.703521:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699233.703523:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699233.703525:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699233.703531:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699233.703536:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699233.703546:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699233.703557:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703664:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703674:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699233.703689:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703697:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699233.703710:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699233.703733:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7144) 00000100:00000040:0.0:1787699233.703773:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699258 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703784:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699233.703789:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699233/real 1787699233] req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703801:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699233.703810:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699233.703823:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699233.703827:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699233.703831:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699233.703839:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438528:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699233.703844:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126438528/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699258 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699233.703853:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699234.213799:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699234.277564:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699234.277564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699234.470335:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699235.237588:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699235.301580:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699235.311900:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699235.493657:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699235.621578:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301809007, at 4301809024 00000100:00000040:3.0:1787699235.621698:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301809007, at 4301809024 00000100:00100000:2.0:1787699236.261701:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699236.325581:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699236.326376:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699236.518145:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699237.285570:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699237.349580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699237.350348:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699237.544413:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699238.309575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699238.374139:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699238.374682:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699238.565743:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787699238.821648:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699238.821662:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787699238.821667:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699238.821680:0:48434:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7139 00000100:00080000:0.0:1787699238.821693:0:48434:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699238.821696:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.821767:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.821772:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:0.0:1787699238.821778:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:0.0:1787699238.821781:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.821784:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.821787:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:0.0:1787699238.821791:0:48434:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:0.0:1787699238.821810:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:0.0:1787699238.821843:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699238.822042:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001c0eba45] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699238.822067:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699238.822072:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787699238.822074:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699238.822079:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7139 00000100:00080000:0.0:1787699238.822083:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699238.822086:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822089:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822092:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:0.0:1787699238.822095:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:0.0:1787699238.822098:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822101:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822103:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:0.0:1787699238.822106:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699238.822117:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699238.822122:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699238.822135:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699238.822145:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000066638187] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699238.822155:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699238.822158:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1787699238.822160:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699238.822164:0:48434:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7139 00000100:00080000:0.0:1787699238.822168:0:48434:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699238.822170:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822173:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822175:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:0.0:1787699238.822178:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:0.0:1787699238.822180:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822183:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822185:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:0.0:1787699238.822188:0:48434:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699238.822192:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:0.0:1787699238.822196:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:0.0:1787699238.822204:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699238.822213:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003dcf9255] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699238.822221:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699238.822225:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787699238.822226:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699238.822230:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7139 00000100:00080000:0.0:1787699238.822234:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699238.822236:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822239:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822241:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:0.0:1787699238.822243:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:0.0:1787699238.822246:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699238.822290:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822307:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126438656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699238.822349:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.822359:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699238.822405:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699238.822442:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822480:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699238/real 1787699238] req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822498:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.822509:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699238.822519:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699238.822524:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699238.822528:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822537:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126438656:192.168.201.136@tcp:250:kworker.0 00000100:00000040:3.0:1787699238.822545:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822554:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699238.822568:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.822577:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699238.822589:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699238.822595:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.822605:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:3.0:1787699238.822611:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699238/real 1787699238] req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822621:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.822630:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699238.822637:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699238.822641:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699238.822645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822653:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a95180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438784:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699238.822659:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.822667:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699238.822679:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.822686:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699238.822695:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822783:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822785:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:0.0:1787699238.822788:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699238.822794:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699238.822799:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699238.822810:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699238.822820:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008f478df6] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699238.822828:0:48434:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787699238.822831:0:48434:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787699238.822833:0:48434:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787699238.822837:0:48434:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7139 00000100:00080000:0.0:1787699238.822840:0:48434:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:0.0:1787699238.822842:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822845:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822846:0:48434:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:0.0:1787699238.822849:0:48434:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699238.822852:0:48434:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699238.822854:0:48434:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:0.0:1787699238.822856:0:48434:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:0.0:1787699238.822859:0:48434:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:0.0:1787699238.822864:0:48434:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:0.0:1787699238.822868:0:48434:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699238.822876:0:48434:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699238.822886:0:48434:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000076e79264] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699238.822895:0:48434:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7149) 00000100:00000040:3.0:1787699238.823110:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823121:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:3.0:1787699238.823128:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699238/real 1787699238] req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823152:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823163:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699238.823172:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699238.823174:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699238.823179:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823187:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126438912:192.168.201.136@tcp:38:kworker.0 00000100:00000040:3.0:1787699238.823193:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94380 x1874531126438656/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699238.823204:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699238.823214:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a95180 x1874531126438784/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699238.823222:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699238.823228:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97b80 x1874531126438912/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699238.823236:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:3.0:1787699238.823522:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823533:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699238.823551:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823557:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699238.823572:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699238.823581:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823591:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:3.0:1787699238.823597:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699238/real 1787699238] req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823609:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823617:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699238.823625:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699238.823628:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699238.823632:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823640:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a94e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439040:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699238.823646:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823655:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699238.823668:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823674:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:3.0:1787699238.823684:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:3.0:1787699238.823690:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699263 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823762:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699238.823769:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699238/real 1787699238] req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823781:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699238.823789:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699238.823797:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787699238.823800:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:3.0:1787699238.823804:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787699238.823819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc9a97100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439168:192.168.201.136@tcp:8:kworker.0 00000100:00000040:3.0:1787699238.823826:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a94e00 x1874531126439040/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699238.823834:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:3.0:1787699238.823840:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc9a97100 x1874531126439168/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699263 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:3.0:1787699238.823847:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699239.333559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699239.398603:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699239.399159:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699239.589747:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699240.357633:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699240.421656:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699240.421754:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699240.613867:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699240.741569:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301814128, at 4301814144 00000100:00000040:3.0:1787699240.741676:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301814128, at 4301814144 00000100:00100000:2.0:1787699241.381808:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699241.445527:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699241.445766:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699241.638099:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699242.405545:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699242.470171:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699242.470210:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699242.661729:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699243.429585:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699243.493654:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699243.494038:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699243.686719:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699243.942497:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699243.942510:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699243.942515:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699243.942528:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7144 00000100:00080000:1.0:1787699243.942935:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699243.942940:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.942945:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.942949:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699243.942954:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699243.942958:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.942969:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.942973:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699243.942976:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699243.942995:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699243.943038:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699243.943104:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000006a8b7364] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699243.943130:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699243.943134:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699243.943136:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699243.943140:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7144 00000100:00080000:1.0:1787699243.943143:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699243.943146:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.943148:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.943151:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699243.943153:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699243.943156:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.943190:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.943193:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699243.943196:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699243.943207:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699243.943214:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699243.943223:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699243.943234:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d5b6fc63] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699243.943243:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699243.943246:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699243.943248:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699243.943252:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7144 00000100:00080000:1.0:1787699243.943256:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699243.943259:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.943261:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.943263:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699243.943265:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699243.943269:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.943271:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.943274:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787699243.943276:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699243.943281:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699243.943285:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699243.943292:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699243.943302:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ddc85e6a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699243.943311:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699243.943315:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699243.943316:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699243.943320:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7144 00000100:00080000:1.0:1787699243.943323:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699243.943326:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.943328:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.943330:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699243.943332:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699243.943335:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.943338:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.943340:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699243.943342:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699243.944668:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699243.944682:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699243.944704:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699243.944715:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000083c933a0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699243.944723:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699243.944727:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699243.944729:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699243.944733:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7144 00000100:00080000:1.0:1787699243.944739:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699243.944742:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.944745:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.944748:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699243.944751:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699243.944754:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699243.944757:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699243.944759:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699243.944761:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699243.944767:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699243.944770:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699243.944778:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699243.944787:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000467430fe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699243.944799:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7155) 00000100:00000040:0.0:1787699243.945250:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.945266:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126439296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699243.945300:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.945308:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699243.945352:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699243.945373:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.945410:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699243/real 1787699243] req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.945460:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.945470:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699243.945479:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699243.945483:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699243.945487:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.945495:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126439296:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699243.945503:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.945512:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699243.945525:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.945532:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699243.945542:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699243.945547:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.945557:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699243.945562:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699243/real 1787699243] req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.945572:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.945580:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699243.945588:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699243.945591:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699243.945594:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946514:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439424:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699243.946527:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946537:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699243.946555:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946562:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699243.946578:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699243.946587:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946597:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699243.946603:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699243/real 1787699243] req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946614:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946622:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699243.946630:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699243.946633:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699243.946637:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946645:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594aa00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439552:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699243.946651:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946660:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699243.946673:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946681:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699243.946690:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699243.946695:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946704:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699243.946709:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699243/real 1787699243] req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946720:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946728:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699243.946736:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699243.946739:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699243.946742:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946748:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439680:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699243.946754:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946762:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699243.946773:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946786:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699243.946797:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699243.946803:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699268 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946812:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699243.946816:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699243/real 1787699243] req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946827:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699243.946835:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699243.946842:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699243.946846:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699243.946849:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699243.946857:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126439808:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699243.946863:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948a80 x1874531126439296/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699243.946874:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699243.946884:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949c00 x1874531126439424/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699243.946890:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699243.946895:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594aa00 x1874531126439552/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699243.946903:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699243.946908:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948700 x1874531126439680/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699243.946914:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699243.946918:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a300 x1874531126439808/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699268 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699243.946924:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699244.453613:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699244.517602:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699244.517718:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699244.709822:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699245.477576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699245.541573:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699245.541675:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699245.733786:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699245.861570:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301819250, at 4301819264 00000100:00000040:2.0:1787699245.861797:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301819250, at 4301819264 00000100:00100000:3.0:1787699246.501750:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699246.565522:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699246.565618:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699246.757669:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699247.525646:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699247.589590:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699247.589771:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699247.781850:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699248.549521:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699248.613611:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699248.613942:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699248.806519:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699249.062025:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.062035:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699249.062038:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.062049:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7150 00000100:00080000:3.0:1787699249.062059:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699249.062062:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.062067:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.062070:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699249.062088:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699249.062093:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.062205:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.062209:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699249.062212:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699249.062226:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699249.062253:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699249.062268:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.062362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.062376:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126439936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699249.062402:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.062411:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699249.062481:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699249.062536:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.062540:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699249.062542:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.062547:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7150 00000100:00080000:3.0:1787699249.062551:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699249.062554:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.062557:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.062559:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699249.062562:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699249.062565:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.062567:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.062569:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699249.062572:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699249.062581:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699249.062586:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699249.062596:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699249.062606:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.062800:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.062834:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.062849:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.062867:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699249.062876:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699249.062881:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699249.062885:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.062893:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126439936:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699249.062900:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126439936/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699249.062911:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699249.062933:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.062942:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699249.062956:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.062964:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699249.062975:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699249.062982:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.062992:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699249.062997:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063008:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.063017:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699249.063029:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699249.063032:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699249.063036:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063043:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440064:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699249.063049:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126440064/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699249.063056:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00080000:3.0:1787699249.063367:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.063372:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699249.063374:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.063379:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7150 00000100:00080000:3.0:1787699249.063383:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699249.063386:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.063389:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.063391:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699249.063394:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699249.063397:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.063399:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.063401:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699249.063404:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699249.063409:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699249.063414:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699249.063445:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699249.063456:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699249.063471:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.063475:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699249.063476:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.063480:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7150 00000100:00080000:3.0:1787699249.063483:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699249.063486:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.063489:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.063491:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699249.063493:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699249.063495:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.063498:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.063499:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699249.063502:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699249.063507:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699249.063510:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699249.063517:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699249.063526:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699249.063534:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699249.063537:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699249.063538:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699249.063542:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7150 00000100:00080000:3.0:1787699249.063545:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699249.063548:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.063550:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.063552:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699249.063554:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699249.063557:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699249.063559:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699249.063561:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699249.063564:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699249.063568:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699249.063571:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699249.063577:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699249.063586:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699249.063595:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7160) 00000100:00000040:0.0:1787699249.063650:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063661:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699249.063675:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.063683:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699249.063697:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699249.063704:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.063715:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699249.063721:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063732:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.063746:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699249.063755:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699249.063757:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699249.063761:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063768:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440192:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699249.063774:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063781:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699249.063792:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.063798:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699249.063807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699249.063813:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.063821:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699249.063825:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063835:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.063844:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699249.063852:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699249.063854:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699249.063972:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063980:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440320:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699249.063985:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.063994:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699249.064006:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.064013:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699249.064024:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699249.064031:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699274 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.064039:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699249.064044:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699249/real 1787699249] req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.064055:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699249.064064:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699249.064072:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699249.064086:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699249.064090:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699249.064162:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440448:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699249.064168:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126440192/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699249.064177:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699249.064183:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126440320/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699249.064190:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699249.064195:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126440448/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699274 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699249.064201:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699249.574586:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699249.637569:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699249.638076:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699249.830114:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699250.597582:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699250.661601:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699250.661761:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699250.853760:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699250.981586:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301824368, at 4301824384 00000100:00000040:3.0:1787699250.985153:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301824368, at 4301824387 00000100:00100000:3.0:1787699251.622022:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699251.685637:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699251.686186:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699251.879156:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699252.646565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699252.709556:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699252.709725:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699252.901755:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699253.669583:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699253.733580:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699253.736938:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699253.925658:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699254.181596:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699254.181613:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699254.181620:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699254.181635:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7155 00000100:00080000:3.0:1787699254.181650:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699254.181655:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.181660:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.181665:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699254.181671:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699254.181675:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.181678:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.181681:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699254.181685:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699254.181704:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699254.181739:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699254.181758:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699254.181840:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699254.181857:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126440576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699254.181893:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699254.181903:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:3.0:1787699254.182036:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699254.182040:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699254.182051:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699254.182057:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7155 00000100:00080000:3.0:1787699254.182062:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699254.182065:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182068:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182071:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699254.182075:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699254.182078:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182080:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182084:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699254.182086:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699254.182099:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699254.182105:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699254.182119:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699254.182129:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699254.182172:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699254.182181:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440704:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699254.182200:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699254.182209:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699254.182225:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00080000:3.0:1787699254.182449:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699254.182453:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699254.182455:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699254.182460:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7155 00000100:00080000:3.0:1787699254.182465:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699254.182468:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182470:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182472:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699254.182475:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699254.182478:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182480:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182482:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699254.182485:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699254.182490:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699254.182496:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699254.182508:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699254.182523:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699254.182541:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787699254.182748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699254.182758:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440832:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699254.182772:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699254.182780:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699254.182795:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00080000:3.0:1787699254.182831:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699254.182833:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699254.182838:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7155 00000100:00080000:3.0:1787699254.182843:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699254.182846:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182849:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182851:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699254.182854:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699254.182857:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182860:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182862:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699254.182865:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699254.182871:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699254.182876:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699254.182886:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699254.182896:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699254.182913:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699254.182917:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699254.182918:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699254.182923:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7155 00000100:00080000:3.0:1787699254.182926:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699254.182929:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182932:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182934:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699254.182936:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699254.182939:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699254.182941:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699254.182943:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699254.182945:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699254.182950:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699254.182954:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699254.182962:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699254.182970:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699254.182980:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7165) 00000100:00000040:0.0:1787699254.183671:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699254.183683:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440960:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699254.183708:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699254.183717:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699254.183739:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00000040:0.0:1787699254.183747:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699254.183755:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699254.183768:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699254.183775:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699254.183787:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:3.0:1787699254.693676:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699254.757689:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699254.757797:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699254.949765:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699255.717824:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699255.781854:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699255.782005:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699255.973767:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699256.103112:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301829487, at 4301829505 00000100:00000040:3.0:1787699256.105070:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301829487, at 4301829507 00000100:00100000:3.0:1787699256.741562:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699256.805539:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699256.805539:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699256.998605:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1787699257.258743:0:2389:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.201.136:988 00000400:00000100:3.0:1787699257.259946:0:2389:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.201.136@tcp at host 192.168.201.136 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787699257.259969:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787699257.260725:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787699257.260752:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787699257.260765:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787699257.260791:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787699257.260799:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787699257.260809:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00100000:0.0:1787699257.261170:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699254/real 1787699257] req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.261197:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699257.261208:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699257.261220:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699257.261225:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699257.261230:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.261239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440832:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787699257.261247:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699254/real 1787699257] req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.261259:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699257.261267:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699257.261274:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699257.261278:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699257.261282:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.261290:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440704:192.168.201.136@tcp:38:kworker.0 00000100:00100000:0.0:1787699257.261298:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699254/real 1787699257] req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.261309:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699257.261316:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699257.261323:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699257.261326:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699257.261330:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.261338:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126440576:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699257.261344:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126440832/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699257.261358:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699257.261369:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126440704/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699257.261376:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699257.261381:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126440576/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699257.261388:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:3.0:1787699257.261500:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787699257.261519:0:2389:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.136@tcp 00000100:00000040:3.0:1787699257.261530:0:2389:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699279 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.262715:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699254/real 1787699257] req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.262743:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699257.262753:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699257.262762:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699257.262767:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699257.262771:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.262951:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126440960:192.168.201.136@tcp:8:kworker.0 00000100:00100000:0.0:1787699257.262961:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699254/real 1787699257] req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.262975:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699257.262983:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699257.263521:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699257.263526:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699257.263532:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699257.263541:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441088:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699257.263548:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126440960/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699257.263563:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699257.263577:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126441088/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699279 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699257.263585:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699257.766880:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699257.829666:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699257.829798:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699258.026706:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699258.789657:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699258.853646:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699258.863653:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699259.045702:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699259.301754:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699259.301766:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699259.301771:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699259.301784:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7160 00000100:00080000:3.0:1787699259.301797:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699259.301811:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.301816:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.301821:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699259.301827:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699259.301833:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.301836:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.301839:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699259.301843:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699259.301863:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699259.301910:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699259.301927:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699259.301958:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699259.301962:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699259.301964:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699259.301969:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7160 00000100:00080000:3.0:1787699259.301973:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699259.301976:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.301979:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.301981:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699259.301984:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699259.301987:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.301989:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.301991:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699259.301994:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699259.302005:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699259.302011:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699259.302022:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699259.302038:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699259.302047:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699259.302051:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699259.302053:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699259.302056:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7160 00000100:00080000:3.0:1787699259.302061:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699259.302064:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.302066:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.302068:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699259.302071:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699259.302073:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.302076:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.302078:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699259.302081:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699259.302086:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699259.302091:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699259.302100:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699259.302110:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699259.302118:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699259.302121:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699259.302124:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699259.302128:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7160 00000100:00080000:3.0:1787699259.302131:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699259.302135:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.302137:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.302139:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699259.302142:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699259.302145:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.302147:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.302150:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699259.302152:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699259.302157:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699259.302161:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699259.302168:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699259.302177:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699259.302185:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699259.302188:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699259.302189:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699259.302193:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7160 00000100:00080000:3.0:1787699259.302196:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699259.302198:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.302201:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.302203:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699259.302205:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699259.302208:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699259.302211:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699259.302214:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699259.302216:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699259.302220:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699259.302224:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699259.302231:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699259.302240:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699259.302252:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7170) 00000100:00000040:0.0:1787699259.303027:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303047:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126441216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699259.303110:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303130:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699259.303195:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699259.303263:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303334:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699259/real 1787699259] req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303363:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303385:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699259.303410:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699259.303416:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699259.303471:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303492:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126441216:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699259.303499:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303520:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699259.303548:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303557:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699259.303586:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699259.303606:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303618:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699259.303629:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699259/real 1787699259] req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303658:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303667:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699259.303687:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699259.303691:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699259.303696:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303705:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441344:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699259.303727:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303757:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699259.303785:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303817:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699259.303845:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699259.303853:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303863:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699259.303869:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699259/real 1787699259] req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.303931:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.303979:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699259.304004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699259.304009:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699259.304033:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.304040:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441472:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699259.304046:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.304068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699259.304083:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.304090:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699259.304102:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699259.304108:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.304118:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699259.304124:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699259/real 1787699259] req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.304135:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.304143:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699259.304150:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699259.304153:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699259.304157:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.304163:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441600:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699259.304169:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.304177:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699259.304190:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.304197:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699259.304209:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699259.304216:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699284 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.304225:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699259.304230:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699259/real 1787699259] req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.304241:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699259.304250:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699259.304258:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699259.304261:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699259.304265:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699259.304272:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441728:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699259.304278:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126441216/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699259.304290:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699259.304303:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126441344/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699259.304309:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699259.304314:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126441472/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699259.304322:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699259.304326:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126441600/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699259.304332:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699259.304337:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126441728/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699284 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699259.304344:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699259.813579:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699259.877557:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699259.877665:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699260.082280:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699260.838032:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699260.901606:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699260.901767:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699261.094115:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699261.221573:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301834608, at 4301834624 00000100:00000040:3.0:1787699261.221573:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301834608, at 4301834624 00000100:00100000:3.0:1787699261.862035:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699261.925585:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699261.925634:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699262.118505:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699262.885892:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699262.949544:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699262.949647:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699263.145787:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699263.909550:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699263.973543:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699263.973543:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699264.165670:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699264.421630:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699264.421644:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699264.421649:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699264.421662:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7165 00000100:00080000:3.0:1787699264.421688:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699264.421692:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421697:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421701:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699264.421708:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699264.421712:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421715:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421719:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699264.421722:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421742:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699264.421774:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699264.421790:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699264.421826:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699264.421830:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699264.421832:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699264.421836:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7165 00000100:00080000:3.0:1787699264.421840:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699264.421843:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421845:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421847:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699264.421850:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699264.421852:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421855:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421857:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699264.421860:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699264.421871:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699264.421876:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699264.421887:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699264.421897:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699264.421906:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699264.421909:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699264.421911:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699264.421915:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7165 00000100:00080000:3.0:1787699264.421918:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699264.421921:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421923:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421925:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699264.421927:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699264.421929:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421932:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421934:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699264.421936:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699264.421940:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699264.421944:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699264.421952:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699264.421960:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699264.421968:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699264.421971:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699264.421973:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699264.421977:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7165 00000100:00080000:3.0:1787699264.421980:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699264.421983:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421985:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421987:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699264.421989:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699264.421992:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.421994:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.421996:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699264.421998:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699264.422003:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699264.422006:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699264.422015:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699264.422023:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699264.422030:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699264.422034:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699264.422035:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699264.422038:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7165 00000100:00080000:3.0:1787699264.422042:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699264.422044:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.422046:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.422048:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699264.422050:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699264.422053:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699264.422055:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699264.422058:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699264.422060:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699264.422064:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699264.422068:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699264.422075:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699264.422083:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699264.422092:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7175) 00000100:00000040:0.0:1787699264.422459:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422473:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126441856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699264.422504:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422512:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699264.422554:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699264.422574:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422614:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699264/real 1787699264] req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422631:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422640:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699264.422648:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699264.422652:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699264.422656:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422663:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126441856:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699264.422682:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422691:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699264.422707:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422713:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699264.422726:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699264.422733:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422745:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699264.422752:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699264/real 1787699264] req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422763:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422774:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699264.422780:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699264.422782:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699264.422786:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422810:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126441984:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699264.422826:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422834:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699264.422849:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422855:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699264.422869:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699264.422876:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422885:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699264.422890:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699264/real 1787699264] req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422899:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422907:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699264.422924:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699264.422927:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699264.422930:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422938:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442112:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699264.422943:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.422951:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699264.422963:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422969:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699264.422979:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699264.422985:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.422994:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699264.422999:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699264/real 1787699264] req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.423009:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.423017:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699264.423024:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699264.423027:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699264.423030:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.423037:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442240:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699264.423042:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.423050:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699264.423061:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.423068:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699264.423076:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699264.423083:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699289 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.423091:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699264.423095:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699264/real 1787699264] req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.423105:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699264.423113:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699264.423119:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699264.423123:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699264.423127:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699264.423133:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442368:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699264.423139:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126441856/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699264.423152:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699264.423163:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126441984/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699264.423170:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699264.423176:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126442112/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699264.423182:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699264.423187:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126442240/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699264.423193:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699264.423198:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126442368/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699289 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699264.423204:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699264.933574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699264.997595:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699264.997775:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699265.190608:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699265.957951:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699266.021508:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699266.021508:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699266.213850:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699266.341605:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301839728, at 4301839744 00000100:00000040:2.0:1787699266.341614:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301839728, at 4301839744 00000100:00100000:0.0:1787699266.982927:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699267.045562:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699267.045615:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699267.237904:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699268.005576:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699268.069564:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699268.070599:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699268.265661:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699269.029874:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699269.095121:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699269.095298:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699269.285698:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699269.543493:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699269.543505:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699269.543511:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699269.543525:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7170 00000100:00080000:2.0:1787699269.543539:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699269.543542:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543547:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543551:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699269.543556:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699269.543559:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543562:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543576:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699269.543580:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543597:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699269.543638:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699269.543656:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000037d8200c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699269.543686:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699269.543690:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699269.543692:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699269.543696:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7170 00000100:00080000:2.0:1787699269.543700:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699269.543703:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543706:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543708:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699269.543711:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699269.543714:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543716:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543718:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699269.543721:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699269.543740:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699269.543746:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699269.543757:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699269.543767:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000008817902a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699269.543781:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699269.543784:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699269.543785:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699269.543789:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7170 00000100:00080000:2.0:1787699269.543792:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699269.543794:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543797:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543799:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699269.543801:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699269.543804:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543807:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543809:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699269.543811:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699269.543815:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699269.543819:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699269.543828:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699269.543837:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000073d3f8a3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699269.543844:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699269.543847:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699269.543848:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699269.543853:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7170 00000100:00080000:2.0:1787699269.543856:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699269.543859:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543861:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543863:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699269.543865:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699269.543868:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.543870:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.543871:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699269.543874:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699269.543879:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699269.543883:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699269.544078:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699269.544097:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699269.544106:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699269.544109:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699269.544111:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699269.544114:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7170 00000100:00080000:2.0:1787699269.544118:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699269.544121:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.544123:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.544125:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699269.544127:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699269.544130:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699269.544132:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699269.544134:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699269.544137:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699269.544143:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699269.544158:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699269.544168:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699269.544177:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699269.544187:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7180) 00000100:00000040:0.0:1787699269.544329:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544346:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126442496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699269.544383:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544393:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699269.544480:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699269.544499:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544545:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699269/real 1787699269] req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544563:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544585:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699269.544594:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699269.544597:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699269.544602:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126442496:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699269.544618:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544637:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699269.544653:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544661:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699269.544674:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699269.544682:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544694:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699269.544700:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699269/real 1787699269] req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544712:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544721:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699269.544741:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699269.544744:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699269.544748:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544756:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442624:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699269.544762:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544771:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699269.544785:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544792:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699269.544804:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699269.544812:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544822:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699269.544828:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699269/real 1787699269] req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544846:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.544854:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699269.544862:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699269.544865:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699269.544869:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.544877:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442752:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699269.544882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.545089:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699269.545106:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.545112:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699269.545125:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699269.545133:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.545144:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699269.545215:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699269/real 1787699269] req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.545236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.545243:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699269.545250:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699269.545254:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699269.545258:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.545265:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126442880:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699269.545270:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.545278:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699269.545290:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.545297:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699269.545309:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699269.545316:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699294 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.545325:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699269.545330:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699269/real 1787699269] req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.545339:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699269.545348:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699269.545355:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699269.545358:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699269.545362:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699269.545369:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443008:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699269.545388:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad880 x1874531126442496/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699269.545400:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699269.545409:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae680 x1874531126442624/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699269.545416:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699269.545448:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad500 x1874531126442752/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699269.545455:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699269.545459:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126442880/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699269.545466:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699269.545471:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126443008/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699294 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699269.545477:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699270.053831:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699270.119515:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699270.120162:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699270.311060:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699271.077960:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699271.141602:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699271.141917:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699271.333922:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699271.461508:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301844850, at 4301844864 00000100:00000040:3.0:1787699271.461675:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301844850, at 4301844864 00000100:00100000:3.0:1787699272.102845:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699272.165594:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699272.165594:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699272.360245:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699273.125540:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699273.191351:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699273.191460:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699273.381729:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787699274.150196:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699274.213813:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699274.214203:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699274.405820:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787699274.661718:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699274.661728:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787699274.661732:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699274.661742:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7175 00000100:00080000:2.0:1787699274.661753:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699274.661756:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.661760:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.661764:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:2.0:1787699274.661769:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:2.0:1787699274.661772:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.661774:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.661777:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:2.0:1787699274.661780:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:2.0:1787699274.661794:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:2.0:1787699274.661820:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699274.661836:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000814d616c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699274.661873:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699274.661877:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787699274.661879:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699274.661882:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7175 00000100:00080000:2.0:1787699274.661886:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699274.661889:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.661891:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.661893:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:2.0:1787699274.661895:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:2.0:1787699274.661898:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.661900:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.661902:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:2.0:1787699274.661904:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699274.661913:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699274.661918:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699274.661926:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699274.661934:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000bd27bb5f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699274.661942:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699274.661944:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1787699274.661946:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699274.661950:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7175 00000100:00080000:2.0:1787699274.661953:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699274.661956:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.661958:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.661960:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:2.0:1787699274.661963:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:2.0:1787699274.661965:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.661968:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.661970:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:2.0:1787699274.661972:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699274.661976:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:2.0:1787699274.661979:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:2.0:1787699274.661987:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699274.661995:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000027bf4c2e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699274.662003:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699274.662007:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787699274.662008:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699274.662012:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7175 00000100:00080000:2.0:1787699274.662015:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699274.662018:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.662021:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.662023:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:2.0:1787699274.662025:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:2.0:1787699274.662027:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.662030:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.662032:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:2.0:1787699274.662034:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699274.662039:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699274.662043:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699274.662051:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699274.662059:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c00597fc] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787699274.662066:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787699274.662068:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787699274.662070:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787699274.662073:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7175 00000100:00080000:2.0:1787699274.662076:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:2.0:1787699274.662079:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.662081:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.662083:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:2.0:1787699274.662085:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:2.0:1787699274.662087:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:2.0:1787699274.662090:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:2.0:1787699274.662092:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:2.0:1787699274.662094:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:2.0:1787699274.662098:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:2.0:1787699274.662103:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699274.662111:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:2.0:1787699274.662120:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5ac3729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1787699274.662129:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7185) 00000100:00000040:0.0:1787699274.662705:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.662720:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126443136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699274.662748:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.662758:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699274.662796:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699274.662814:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.662852:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699274/real 1787699274] req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.662869:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.662878:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699274.662888:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699274.662891:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699274.662895:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.662903:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126443136:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699274.662910:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.662920:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699274.662933:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.662941:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699274.662952:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699274.662960:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.662970:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699274.662976:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699274/real 1787699274] req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.662987:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.662996:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699274.663004:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699274.663007:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699274.663011:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663018:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ad180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443264:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699274.663024:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663032:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699274.663043:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663050:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699274.663061:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699274.663068:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663078:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699274.663083:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699274/real 1787699274] req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663095:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663104:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699274.663111:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699274.663115:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699274.663119:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663127:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ae300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443392:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699274.663133:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663141:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699274.663153:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663160:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699274.663170:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699274.663176:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663192:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699274.663197:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699274/real 1787699274] req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663208:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699274.663225:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699274.663227:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699274.663231:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663239:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35ac380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443520:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699274.663245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663253:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699274.663263:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663271:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699274.663281:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699274.663287:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699299 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663296:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699274.663302:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699274/real 1787699274] req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663313:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699274.663323:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699274.663330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699274.663333:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699274.663337:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699274.663344:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc35af100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443648:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699274.663350:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af800 x1874531126443136/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699274.663361:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699274.663371:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ad180 x1874531126443264/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699274.663379:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699274.663384:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ae300 x1874531126443392/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699274.663391:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699274.663396:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35ac380 x1874531126443520/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699274.663404:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699274.663410:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc35af100 x1874531126443648/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699299 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699274.663417:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699275.174113:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699275.237549:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699275.237667:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699275.429692:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699276.202639:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699276.261684:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699276.262280:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699276.456244:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699276.581544:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301849968, at 4301849984 00000100:00000040:3.0:1787699276.581564:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301849968, at 4301849984 00000100:00100000:0.0:1787699277.224404:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699277.285550:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699277.285711:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699277.477934:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699278.245635:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699278.309526:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699278.309526:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699278.501765:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699279.272057:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699279.333535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699279.337991:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699279.526699:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699279.781610:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699279.781621:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699279.781624:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699279.781634:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7180 00000100:00080000:3.0:1787699279.781646:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699279.781649:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781652:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781656:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699279.781661:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699279.781664:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781676:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781679:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699279.781682:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781696:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699279.781720:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699279.781734:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003f37ea52] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699279.781761:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699279.781765:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699279.781766:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699279.781770:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7180 00000100:00080000:3.0:1787699279.781774:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699279.781777:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781779:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781781:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699279.781784:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699279.781787:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781789:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781791:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699279.781793:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699279.781802:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699279.781807:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699279.781817:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699279.781825:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ed3f5ae9] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699279.781833:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699279.781836:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699279.781837:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699279.781841:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7180 00000100:00080000:3.0:1787699279.781845:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699279.781848:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781850:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781852:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699279.781855:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699279.781857:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781860:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781862:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699279.781864:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699279.781869:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699279.781872:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699279.781881:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699279.781889:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000aeacd729] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699279.781896:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699279.781898:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699279.781900:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699279.781903:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7180 00000100:00080000:3.0:1787699279.781907:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699279.781909:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781911:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781914:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699279.781917:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699279.781919:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781922:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781924:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699279.781926:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699279.781931:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699279.781935:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699279.781943:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699279.781952:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ae502cd0] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699279.781960:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699279.781963:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699279.781965:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699279.781968:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7180 00000100:00080000:3.0:1787699279.781971:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699279.781974:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781976:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781979:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699279.781980:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699279.781983:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699279.781986:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699279.781987:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699279.781990:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699279.781995:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699279.781999:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699279.782007:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699279.782016:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000007494333f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699279.782026:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7190) 00000100:00000040:0.0:1787699279.782463:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782476:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126443776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699279.782503:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782511:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699279.782568:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699279.782585:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782620:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699279/real 1787699279] req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782635:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782645:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699279.782654:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699279.782657:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699279.782661:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126443776:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699279.782689:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782696:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699279.782710:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782717:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699279.782728:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699279.782735:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782744:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699279.782749:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699279/real 1787699279] req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782759:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782766:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699279.782773:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699279.782776:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699279.782779:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782786:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126443904:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699279.782791:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782800:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699279.782812:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782819:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699279.782830:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699279.782836:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782846:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699279.782851:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699279/real 1787699279] req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782860:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782867:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699279.782874:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699279.782877:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699279.782881:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782889:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444032:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699279.782894:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782902:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699279.782914:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782921:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699279.782933:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699279.782941:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782951:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699279.782956:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699279/real 1787699279] req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782966:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.782973:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699279.782980:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699279.782982:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699279.782985:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.782992:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444160:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699279.782997:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.783005:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699279.783018:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.783025:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699279.783038:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699279.783045:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699304 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.783055:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699279.783060:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699279/real 1787699279] req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.783071:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699279.783079:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699279.783087:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699279.783090:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699279.783093:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699279.783111:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444288:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699279.783118:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3480 x1874531126443776/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699279.783128:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699279.783137:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1500 x1874531126443904/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699279.783149:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699279.783153:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2300 x1874531126444032/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699279.783159:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699279.783164:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0380 x1874531126444160/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699279.783171:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699279.783175:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3100 x1874531126444288/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699304 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699279.783181:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699280.293525:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699280.357535:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699280.357535:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699280.549578:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699281.317725:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699281.381548:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699281.381653:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699281.574411:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699281.701515:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301855088, at 4301855104 00000100:00000040:3.0:1787699281.701515:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301855088, at 4301855104 00000100:00100000:0.0:1787699282.341666:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699282.405611:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699282.405621:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699282.597680:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699283.366978:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699283.430470:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699283.430605:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699283.621773:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699284.389554:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699284.453512:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699284.454815:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699284.645673:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699284.901867:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699284.901880:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699284.901885:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699284.901898:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7185 00000100:00080000:1.0:1787699284.901912:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699284.901916:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.901921:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.901925:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699284.901931:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699284.901934:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.901937:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.901940:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699284.901944:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699284.901963:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699284.901996:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699284.902013:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000c0b91c73] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699284.902038:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699284.902042:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699284.902044:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699284.902049:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7185 00000100:00000040:0.0:1787699284.902116:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699284.902120:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699284.902133:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.902140:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.902144:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000100:00100000:0.0:1787699284.902145:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126444416:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787699284.902155:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699284.902174:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.902179:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699284.902182:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699284.902190:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:0.0:1787699284.902205:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699284.902205:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699284.902224:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699284.902234:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000800:00000100:0.0:1787699284.902242:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787699284.902251:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699284.902261:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699284.902274:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000082ca088b] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699284.902288:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699284.902292:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699284.902294:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699284.902302:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7185 00000100:00100000:0.0:1787699284.902304:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699284/real 1787699284] req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699284.902312:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699284.902321:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787699284.902325:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699284.902328:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.902335:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699284.902340:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699284.902350:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.902358:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000100:00000040:0.0:1787699284.902359:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:1.0:1787699284.902362:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787699284.902370:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:0.0:1787699284.902386:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787699284.902386:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787699284.902395:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787699284.902398:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699284.902400:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.902414:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126444416:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699284.902489:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948000 x1874531126444416/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699284.902501:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699284.902520:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.902526:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699284.902538:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.902543:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699284.902552:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699284.902557:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.902565:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699284.902569:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699284/real 1787699284] req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.902580:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.902588:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699284.902596:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699284.902599:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699284.902603:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.902610:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444544:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699284.902616:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949500 x1874531126444544/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699284.902622:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699284.902785:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699284.902799:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d8ddb79c] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699284.902821:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699284.902825:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699284.902827:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699284.902832:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7185 00000100:00080000:1.0:1787699284.902837:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699284.902840:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.902844:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.902847:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699284.902851:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699284.902854:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.902858:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.902865:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00000040:0.0:1787699284.902867:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699284.902872:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699284.902883:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699284.902891:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699284.902892:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444672:192.168.201.136@tcp:38:kworker.0 00000100:00100000:1.0:1787699284.902910:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:0.0:1787699284.902916:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699284.902944:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000001675000f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.902946:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699284.902962:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699284.902970:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787699284.902971:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699284.902975:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699284.902985:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7185 00000100:00000040:0.0:1787699284.902989:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699284.902993:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699284.903003:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699284.903009:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.903012:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699284.903016:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699284.903017:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00000040:1.0:1787699284.903024:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00100000:0.0:1787699284.903032:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699284/real 1787699284] req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699284.903033:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699284.903040:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699284.903051:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00100000:0.0:1787699284.903061:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.903070:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699284.903077:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699284.903081:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699284.903085:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.903092:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444672:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699284.903098:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b480 x1874531126444672/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699284.903106:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699284.903116:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787699284.903131:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1787699284.903132:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444800:192.168.201.136@tcp:8:kworker.0 00000020:00000040:1.0:1787699284.903143:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699284.903156:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.903165:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00100000:1.0:1787699284.903173:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699284.903184:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699284.903196:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dbb8239a] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.903199:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699284.903248:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7196) 00000100:00000040:0.0:1787699284.903265:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699284.903274:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699284/real 1787699284] req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.903286:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.903294:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699284.903302:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699284.903305:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699284.903308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.903316:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444800:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699284.903322:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b100 x1874531126444800/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699284.903329:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699284.903338:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.903347:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699284.903357:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.903364:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699284.903374:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699284.903380:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699309 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.903389:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699284.903394:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699284/real 1787699284] req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.903404:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699284.903412:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699284.903437:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699284.903441:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699284.903446:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699284.903453:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594ad80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126444928:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699284.903458:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594ad80 x1874531126444928/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699309 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699284.903476:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699285.413574:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:1.0:1787699285.477597:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699285.478026:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699285.669589:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699286.438320:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699286.501593:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699286.501594:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699286.693680:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699286.821625:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301860207, at 4301860224 00000100:00000040:3.0:1787699286.821707:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301860207, at 4301860224 00000100:00100000:0.0:1787699287.461559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699287.525606:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699287.525835:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699287.718462:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699288.486549:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699288.549613:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699288.549614:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699288.742241:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699289.509526:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699289.573975:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699289.576653:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699289.772448:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787699290.024002:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699290.024016:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787699290.024021:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699290.024034:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7191 00000100:00080000:3.0:1787699290.024048:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699290.024051:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024055:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024059:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:3.0:1787699290.024064:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:3.0:1787699290.024066:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024068:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024071:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:3.0:1787699290.024073:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024092:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:3.0:1787699290.024127:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699290.024142:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000002eac7cf2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.024171:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699290.024176:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787699290.024178:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699290.024183:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7191 00000100:00080000:3.0:1787699290.024187:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699290.024190:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024193:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024194:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:3.0:1787699290.024197:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:3.0:1787699290.024199:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024201:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024204:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:3.0:1787699290.024206:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699290.024218:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699290.024223:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699290.024232:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699290.024242:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000a5c602c3] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.024250:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699290.024253:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1787699290.024255:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699290.024259:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7191 00000100:00080000:3.0:1787699290.024262:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699290.024265:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024267:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024270:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:3.0:1787699290.024273:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:3.0:1787699290.024275:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024278:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024280:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:3.0:1787699290.024282:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699290.024287:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:3.0:1787699290.024291:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:3.0:1787699290.024299:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699290.024308:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000000f09b84f] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.024316:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699290.024320:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787699290.024322:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699290.024326:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7191 00000100:00080000:3.0:1787699290.024329:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699290.024332:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024334:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024336:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:3.0:1787699290.024337:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:3.0:1787699290.024339:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024341:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024342:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:3.0:1787699290.024344:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699290.024348:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699290.024351:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699290.024359:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699290.024369:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000004ac6036e] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787699290.024377:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787699290.024389:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787699290.024391:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787699290.024395:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7191 00000100:00080000:3.0:1787699290.024399:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:3.0:1787699290.024401:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024404:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024406:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:3.0:1787699290.024408:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:3.0:1787699290.024411:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:3.0:1787699290.024413:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:3.0:1787699290.024415:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:3.0:1787699290.024436:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:3.0:1787699290.024441:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:3.0:1787699290.024446:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:3.0:1787699290.024454:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:3.0:1787699290.024464:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000510600d5] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1787699290.024474:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7201) 00000100:00000040:0.0:1787699290.026676:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.026702:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126445056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699290.026746:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.026757:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699290.026807:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699290.026830:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.026879:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.026896:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.026906:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699290.026917:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699290.026921:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699290.026927:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.026935:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126445056:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699290.026942:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.026950:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699290.026967:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.026974:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699290.026985:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699290.026992:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027004:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699290.027010:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027021:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027031:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699290.027039:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699290.027042:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699290.027046:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027053:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445184:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699290.027060:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027068:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699290.027079:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027087:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699290.027095:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699290.027101:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027110:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699290.027116:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027127:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027136:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699290.027144:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699290.027147:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699290.027151:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027159:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445312:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699290.027165:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027173:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699290.027186:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027193:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699290.027204:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699290.027210:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027220:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699290.027225:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027236:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027245:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699290.027252:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699290.027255:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699290.027259:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027267:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445440:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699290.027274:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027282:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699290.027294:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027302:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699290.027311:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699290.027318:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699315 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027333:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699290.027338:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699290/real 1787699290] req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.027349:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699290.027357:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699290.027364:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699290.027367:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699290.027371:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699290.028819:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc49d3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445568:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699290.028844:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2a00 x1874531126445056/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699290.028861:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699290.028874:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d2d80 x1874531126445184/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699290.028882:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699290.028888:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d0a80 x1874531126445312/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699290.028895:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699290.028900:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d1c00 x1874531126445440/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699290.028909:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699290.028914:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc49d3b80 x1874531126445568/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699315 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699290.028922:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699290.533564:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699290.597565:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699290.597565:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699290.789627:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699291.557508:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699291.621555:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699291.621555:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699291.813680:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1787699291.941552:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301865332, at 4301865344 00000100:00000040:3.0:1787699291.941552:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301865332, at 4301865344 00000100:00100000:0.0:1787699292.581771:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699292.645560:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699292.645560:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699292.837694:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699293.605558:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699293.670473:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699293.671600:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699293.864603:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787699294.629850:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699294.693611:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699294.694436:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699294.887623:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699295.143742:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699295.143758:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699295.143764:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699295.143779:0:86212:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7196 00000100:00080000:1.0:1787699295.143798:0:86212:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699295.143803:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.143808:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.143812:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699295.143819:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699295.143822:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.143825:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.143829:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699295.143834:0:86212:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699295.143858:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699295.143902:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699295.143925:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e0439a17] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.143954:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699295.143959:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699295.143961:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699295.143966:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7196 00000100:00080000:1.0:1787699295.143970:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699295.143973:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.143976:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.143979:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699295.143982:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699295.143986:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.143999:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.144004:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00000040:0.0:1787699295.144006:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.144009:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699295.144024:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787699295.144029:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126445696:192.168.201.136@tcp:250:kworker.0 00000020:00000040:1.0:1787699295.144031:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699295.144045:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699295.144056:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000ee07f96d] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.144067:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000040:0.0:1787699295.144069:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.144073:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699295.144078:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699295.144085:0:86212:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7196 00000100:00000040:0.0:1787699295.144088:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000100:00080000:1.0:1787699295.144092:0:86212:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699295.144096:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144099:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.144101:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699295.144104:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699295.144107:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144110:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.144112:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787699295.144114:0:86212:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699295.144120:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699295.144125:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699295.144134:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787699295.144143:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144151:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000f25509d2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.144160:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699295.144164:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699295.144166:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00000040:0.0:1787699295.144171:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.144172:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7196 00000100:00080000:1.0:1787699295.144179:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699295.144185:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144189:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.144191:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699295.144194:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699295.144198:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144200:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.144202:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699295.144204:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699295.144209:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699295.144214:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699295.144224:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.144225:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699295.144252:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144263:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699295.144263:0:86212:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699295.144270:0:86212:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699295.144271:0:86212:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699295.144276:0:86212:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7196 00000100:00080000:1.0:1787699295.144279:0:86212:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699295.144282:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144288:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.144303:0:86212:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699295.144306:0:86212:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:0.0:1787699295.144308:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699295.144311:0:86212:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144316:0:86212:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699295.144321:0:86212:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699295.144326:0:86212:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000100:00080000:0.0:1787699295.144330:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787699295.144332:0:86212:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1787699295.144339:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000020:00000040:1.0:1787699295.144345:0:86212:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699295.144346:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787699295.144359:0:86212:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787699295.144360:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594bb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126445696:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699295.144375:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594bb80 x1874531126445696/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699295.144378:0:86212:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699295.144399:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:1.0:1787699295.144399:0:86212:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7206) 00000100:00000040:0.0:1787699295.144453:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144462:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699295.144478:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144485:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699295.144498:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699295.144505:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144517:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699295.144522:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144538:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144547:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699295.144554:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699295.144557:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699295.144560:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144566:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445824:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699295.144572:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144579:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699295.144591:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144597:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699295.144608:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699295.144616:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144626:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699295.144632:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144650:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144659:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699295.144667:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699295.144671:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699295.144675:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144682:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594b800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126445952:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699295.144688:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144701:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699295.144715:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144723:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699295.144733:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699295.144740:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144749:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699295.144754:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144769:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144777:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699295.144784:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699295.144787:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699295.144790:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144797:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446080:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699295.144802:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144810:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699295.144821:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144829:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699295.144840:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699295.144847:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699320 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144857:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699295.144864:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699295/real 1787699295] req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144874:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699295.144882:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699295.144890:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699295.144893:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699295.144897:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699295.144905:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446208:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699295.144909:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949880 x1874531126445824/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699295.144917:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699295.144924:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594b800 x1874531126445952/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699295.144930:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699295.144934:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126446080/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699295.144941:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699295.144946:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126446208/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699320 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699295.144952:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:0.0:1787699295.653565:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699295.717823:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699295.718379:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699295.910415:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699296.677575:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699296.741678:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:3.0:1787699296.741724:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699296.933678:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:3.0:1787699297.061560:0:88616:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829b30 (cpu 3) has reached its deadline 4301870449, at 4301870464 00000100:00000040:2.0:1787699297.061562:0:88537:0:(connection.c:75:cpu_latency_work()) work item of ffff9c5cc8829a20 (cpu 2) has reached its deadline 4301870449, at 4301870464 00000100:00100000:2.0:1787699297.701542:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:3.0:1787699297.765551:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699297.765683:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699297.957690:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699298.725534:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699298.789512:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00000100:00100000:0.0:1787699298.790392:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699298.981661:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787699299.749636:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699299.815011:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699299.815221:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699300.005665:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787699300.265328:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) ab07ff44-f6d6-40b1-8c62-193eef24c564->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699300.265339:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787699300.265343:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699300.265353:0:25237:0:(import.c:510:import_select_connection()) MGC192.168.201.136@tcp: connect to NID 192.168.201.136@tcp last attempt 7201 00000100:00080000:1.0:1787699300.265364:0:25237:0:(import.c:589:import_select_connection()) MGC192.168.201.136@tcp: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699300.265367:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265371:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265375:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa469f9 00000020:00000040:1.0:1787699300.265381:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cc2f50800 refcount=3 00000100:00000040:1.0:1787699300.265384:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265387:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265389:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cc2f50800 : new refcount 2 00000100:00080000:1.0:1787699300.265392:0:25237:0:(import.c:624:import_select_connection()) MGC192.168.201.136@tcp: import ffff9c5ccafb6800 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265407:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5ccafb6800 refcount=4 obd=MGC192.168.201.136@tcp 00000100:00100000:1.0:1787699300.265465:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699300.265481:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000dcd7f0db] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699300.265497:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699300.265500:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787699300.265502:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699300.265507:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7201 00000100:00080000:1.0:1787699300.265511:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699300.265514:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265517:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265519:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a4d 00000020:00000040:1.0:1787699300.265521:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930e000 refcount=5 00000100:00000040:1.0:1787699300.265524:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265526:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265529:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930e000 : new refcount 4 00000100:00080000:1.0:1787699300.265531:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff9c5cc7143000: import ffff9c5cc2e98000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699300.265542:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699300.265547:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e98000 refcount=6 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699300.265556:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699300.265566:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000054172071] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699300.265575:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699300.265578:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1787699300.265580:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699300.265583:0:25237:0:(import.c:510:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7201 00000100:00080000:1.0:1787699300.265587:0:25237:0:(import.c:589:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699300.265589:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265591:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265593:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a54 00000020:00000040:1.0:1787699300.265596:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5cf930f000 refcount=5 00000100:00000040:1.0:1787699300.265598:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265601:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265603:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5cf930f000 : new refcount 4 00000100:00080000:1.0:1787699300.265605:0:25237:0:(import.c:624:import_select_connection()) lustre-MDT0001-mdc-ffff9c5cc7143000: import ffff9c5cc3fd3000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699300.265610:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683216 ocd_grant: 2097152, lost: 0. 00000020:00000040:1.0:1787699300.265614:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc3fd3000 refcount=11 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00100000:1.0:1787699300.265621:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699300.265629:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000026a21afe] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699300.265637:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699300.265640:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787699300.265641:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699300.265645:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7201 00000100:00080000:1.0:1787699300.265648:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699300.265651:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265653:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265655:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a62 00000020:00000040:1.0:1787699300.265657:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb6000 refcount=4 00000100:00000040:1.0:1787699300.265660:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265663:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265664:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb6000 : new refcount 3 00000100:00080000:1.0:1787699300.265667:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff9c5cc7143000: import ffff9c5cc2e97000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699300.265671:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699300.265677:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cc2e97000 refcount=5 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699300.265685:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699300.265694:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000098664c36] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787699300.265701:0:25237:0:(pinger.c:229:ptlrpc_pinger_process_import()) eafa5a76-4ab4-4b5b-9cef-e54b33255124->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787699300.265704:0:25237:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787699300.265706:0:25237:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787699300.265709:0:25237:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: connect to NID 192.168.201.136@tcp last attempt 7201 00000100:00080000:1.0:1787699300.265712:0:25237:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: tried all connections, increasing latency to 20s 00000100:00000040:1.0:1787699300.265714:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265717:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265719:0:25237:0:(genops.c:804:class_conn2export()) looking for export cookie 0x3f46ccbc9aa46a69 00000020:00000040:1.0:1787699300.265721:0:25237:0:(lustre_handles.c:148:class_handle2object()) GET export ffff9c5ccafb0000 refcount=4 00000100:00000040:1.0:1787699300.265723:0:25237:0:(lustre_net.h:1965:ptlrpc_connection_put()) PUT conn=ffff9c5cc2dafe00 refcount 14 to 192.168.201.136@tcp 00000100:00000040:1.0:1787699300.265726:0:25237:0:(connection.c:161:ptlrpc_connection_addref()) conn=ffff9c5cc2dafe00 refcount 15 to 192.168.201.136@tcp 00000020:00000040:1.0:1787699300.265728:0:25237:0:(genops.c:879:class_export_put()) PUTting export ffff9c5ccafb0000 : new refcount 3 00000100:00080000:1.0:1787699300.265730:0:25237:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff9c5cc7143000: import ffff9c5cf930c000 using connection 192.168.201.136@tcp/192.168.201.136@tcp 00000008:00100000:1.0:1787699300.265735:0:25237:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683216 ocd_grant: 8388608, lost: 0. 00000020:00000040:1.0:1787699300.265739:0:25237:0:(genops.c:1210:class_import_get()) import ffff9c5cf930c000 refcount=3 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:1.0:1787699300.265747:0:25237:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000040:1.0:1787699300.265755:0:25237:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000500caeb2] to pc [ptlrpcd_rcv+-1] req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1787699300.265766:0:25237:0:(pinger.c:304:ptlrpc_pinger_main()) next wakeup in 5 (7211) 00000100:00000040:0.0:1787699300.265903:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.265917:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126446336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699300.265945:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.265954:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699300.265988:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699300.266005:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266042:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699300/real 1787699300] req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266058:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266067:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699300.266075:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5ccafb6800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699300.266078:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699300.266082:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266091:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:ab07ff44-f6d6-40b1-8c62-193eef24c564:2398:1874531126446336:192.168.201.136@tcp:250:kworker.0 00000100:00000040:0.0:1787699300.266097:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266106:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699300.266121:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266128:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699300.266139:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699300.266158:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266169:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9c5cc7143000: using a timeout of 12876 usecs (13 jiffies) 00000100:00100000:0.0:1787699300.266175:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699300/real 1787699300] req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266185:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266193:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699300.266200:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e98000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699300.266203:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699300.266217:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266224:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446464:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699300.266229:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266237:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699300.266249:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266256:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699300.266267:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699300.266274:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266283:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9c5cc7143000: using a timeout of 15723 usecs (16 jiffies) 00000100:00100000:0.0:1787699300.266288:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699300/real 1787699300] req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266298:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266305:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699300.266313:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc3fd3000 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699300.266315:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699300.266319:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266326:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5949180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446592:192.168.201.136@tcp:38:kworker.0 00000100:00000040:0.0:1787699300.266331:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266338:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699300.266350:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266357:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699300.266368:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699300.266375:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266385:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0000-osc-ffff9c5cc7143000: using a timeout of 19001 usecs (20 jiffies) 00000100:00100000:0.0:1787699300.266391:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699300/real 1787699300] req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266401:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266408:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699300.266414:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cc2e97000 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699300.266417:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699300.266449:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266457:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc594a680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446720:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699300.266462:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266470:0:2398:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699300.266481:0:2398:0:(niobuf.c:1117:ptl_send_rpc()) @@@ send flags=200 req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266488:0:2398:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.201.136@tcp 00000800:00000100:0.0:1787699300.266499:0:2398:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.136@tcp 00000100:00000040:0.0:1787699300.266506:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 1 req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 0 dl 1787699325 ref 2 fl Rpc:eNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266516:0:2398:0:(niobuf.c:635:kick_cpu_latency()) lustre-OST0001-osc-ffff9c5cc7143000: using a timeout of 28167 usecs (29 jiffies) 00000100:00100000:0.0:1787699300.266521:0:2398:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787699300/real 1787699300] req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266531:0:2398:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000040:0.0:1787699300.266539:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Rpc:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787699300.266545:0:2398:0:(import.c:65:import_set_state_nolock()) ffff9c5cf930c000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787699300.266548:0:2398:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.136@tcp failed (-110) 00000100:00000040:0.0:1787699300.266551:0:2398:0:(lustre_net.h:2405:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Interpret:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787699300.266559:0:2398:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff9c5cc5948380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:eafa5a76-4ab4-4b5b-9cef-e54b33255124:2398:1874531126446848:192.168.201.136@tcp:8:kworker.0 00000100:00000040:0.0:1787699300.266564:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948e00 x1874531126446336/t0(0) o250->MGC192.168.201.136@tcp@192.168.201.136@tcp:26/25 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699300.266573:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5ccafb6800 refcount=3 obd=MGC192.168.201.136@tcp 00000100:00000040:0.0:1787699300.266583:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949f80 x1874531126446464/t0(0) o38->lustre-MDT0000-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699300.266588:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e98000 refcount=5 obd=lustre-MDT0000-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699300.266593:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5949180 x1874531126446592/t0(0) o38->lustre-MDT0001-mdc-ffff9c5cc7143000@192.168.201.136@tcp:12/10 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699300.266599:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc3fd3000 refcount=10 obd=lustre-MDT0001-mdc-ffff9c5cc7143000 00000100:00000040:0.0:1787699300.266604:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc594a680 x1874531126446720/t0(0) o8->lustre-OST0000-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699300.266611:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cc2e97000 refcount=4 obd=lustre-OST0000-osc-ffff9c5cc7143000 00000100:00000040:0.0:1787699300.266616:0:2398:0:(client.c:2875:__ptlrpc_req_put()) @@@ refcount now 0 req@ffff9c5cc5948380 x1874531126446848/t0(0) o8->lustre-OST0001-osc-ffff9c5cc7143000@192.168.201.136@tcp:28/4 lens 520/544 e 0 to 1 dl 1787699325 ref 1 fl Complete:EeXNQU/200/ffffffff rc -110/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000020:00000040:0.0:1787699300.266622:0:2398:0:(genops.c:1223:class_import_put()) import ffff9c5cf930c000 refcount=2 obd=lustre-OST0001-osc-ffff9c5cc7143000 00000100:00100000:2.0:1787699300.773559:0:87781:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cc3c80c00 going to sleep for 0 seconds 00000100:00100000:0.0:1787699300.837595:0:87873:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5cd88786c0 going to sleep for 0 seconds 00000100:00100000:1.0:1787699300.838366:0:87839:0:(client.c:2715:ptlrpc_set_wait()) set ffff9c5ce132d3c0 going to sleep for 0 seconds 00010000:00010000:2.0:1787699301.030301:0:87782:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) Debug log: 80717 lines, 80717 kept, 0 dropped, 0 bad.